Nov 02 01:28:58.067065 ip-10-0-135-33 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 01:28:58.067079 ip-10-0-135-33 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 01:28:58.067088 ip-10-0-135-33 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 01:28:58.067418 ip-10-0-135-33 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 01:29:08.316055 ip-10-0-135-33 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 01:29:08.316073 ip-10-0-135-33 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3c7b2f7106d142efa4533db537f35bf3 --
Nov 02 01:31:03.239445 ip-10-0-135-33 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:31:03.564208 ip-10-0-135-33 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:31:03.564208 ip-10-0-135-33 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:31:03.564208 ip-10-0-135-33 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:31:03.564208 ip-10-0-135-33 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:31:03.570445 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570355    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:03.570445 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570442    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:03.570445 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570446    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:03.570445 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570449    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:03.570445 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570452    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:03.570445 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570455    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570458    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570461    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570463    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570466    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570469    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570471    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570474    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570477    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570479    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570482    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570484    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570487    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570490    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570492    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570496    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570499    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570501    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570504    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570507    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:03.570658 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570509    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570513    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570521    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570524    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570526    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570529    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570531    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570534    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570536    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570539    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570542    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570545    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570547    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570550    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570553    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570556    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570559    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570562    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570564    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:03.571121 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570567    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570570    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570572    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570575    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570577    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570580    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570583    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570585    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570588    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570591    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570593    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570596    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570598    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570601    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570604    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570608    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570612    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570615    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570618    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:03.571594 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570621    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570623    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570626    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570629    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570632    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570634    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570637    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570639    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570642    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570645    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570648    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570650    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570653    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570656    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570658    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570661    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570663    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570666    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570669    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:03.572042 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570672    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570676    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570678    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570681    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570684    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570686    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570689    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570692    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570696    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570699    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570702    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570705    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570708    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570710    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570713    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570715    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570719    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570722    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570724    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:03.572513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570726    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570729    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570733    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570735    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.570738    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571573    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571580    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571583    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571586    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571589    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571591    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571594    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571597    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571600    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571602    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571605    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571608    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571612    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571614    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571617    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:03.572967 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571620    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571622    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571625    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571627    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571630    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571632    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571635    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571637    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571640    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571643    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571646    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571649    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571651    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571654    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571657    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571660    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571662    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571665    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571667    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:03.573468 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571670    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571674    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571678    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571681    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571685    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571687    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571690    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571692    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571695    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571698    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571700    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571703    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571705    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571708    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571710    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571713    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571715    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571718    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571720    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571722    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:03.573921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571725    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571727    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571730    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571733    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571737    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571740    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571743    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571746    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571748    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571751    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571754    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571756    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571759    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571761    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571763    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571766    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571769    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571772    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571774    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:03.574412 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571777    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571779    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571782    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571785    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571788    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571790    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571793    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571795    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571798    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571800    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571803    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571806    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571808    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571811    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571814    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571817    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571819    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571822    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571824    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:03.574865 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571827    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571829    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571832    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571834    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571837    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571839    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571842    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571844    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571847    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571849    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571852    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571870    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571878    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.571882    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572462    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572469    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572476    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572480    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572483    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:03.575357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572486    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572488    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572492    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572494    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572497    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572500    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572502    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572505    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572507    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572510    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572513    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572515    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572518    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572520    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572524    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572526    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572529    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572531    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572534    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:03.575804 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572537    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572540    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572542    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572545    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572547    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572550    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572555    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572557    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572560    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572563    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572566    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572570    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572574    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572577    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572579    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572582    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572585    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572587    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572590    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572592    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:03.576269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572596    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572598    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572601    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572603    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572606    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572608    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572611    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572613    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572616    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572619    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572621    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572624    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572626    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572629    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572631    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572633    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572636    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572638    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572652    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572655    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:03.576784 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572658    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572660    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572663    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572665    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572669    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572671    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572674    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572677    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572679    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572682    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572684    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572687    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572689    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572692    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572694    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572697    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572699    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572701    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572706    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:03.577269 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572709    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572712    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572715    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572718    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572721    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572724    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572727    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572730    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572732    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572735    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572737    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572741    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572744    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572746    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572749    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572751    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572754    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572757    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572760    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572762    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:03.577725 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572765    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572767    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.572771    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573745    2573 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573755    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573761    2573 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573767    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573771    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573775    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573779    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573783    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573786    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573790    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573799    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573803    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573806    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573809    2573 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573812    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573815    2573 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573818    2573 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573821    2573 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573825    2573 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573829    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:31:03.578294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573832    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573835    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573839    2573 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573847    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573850    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573853    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573857    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573860    2573 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573863    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573866    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573870    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573872    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573877    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573880    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573882    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573885    2573 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573888    2573 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573891    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573895    2573 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573915    2573 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573918    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573924    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573927    2573 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573932    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:31:03.578936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573935    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573938    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573942    2573 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573945    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573947    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573951    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573953    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573957    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573959    2573 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573962    2573 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573967    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573970    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573973    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573977    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573980    2573 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573983    2573 flags.go:64] FLAG: --help="false"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573986    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573990    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573993    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573996    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.573999    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574002    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574005    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:31:03.579562 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574008    2573 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574011    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574014    2573 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574017    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574020    2573 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574023    2573 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574028    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574030    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574033    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574036    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574039    2573 flags.go:64] FLAG: --lock-file=""
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574042    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574045    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574048    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574053    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574056    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574060    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574062    2573 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574065    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574068    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574071    2573 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574074    2573 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574079    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574082    2573 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574086    2573 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:31:03.580104 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574089    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574092    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574095    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574098    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574101    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574104    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574107    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574114    2573 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574118    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574120    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574124    2573 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574127    2573 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574130    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574134    2573 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574137    2573 flags.go:64] FLAG: --port="10250"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574140    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574143    2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b3b39b36fb2b20e6"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574146    2573 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574149    2573 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574164    2573 flags.go:64] FLAG: --register-node="true"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574168    2573 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574172    2573 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574175    2573 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574178    2573 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574181    2573 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574185    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:31:03.580753 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574188    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574191    2573 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574194    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574197    2573 flags.go:64] FLAG: --runonce="false"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574200    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574203    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574216    2573 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574220    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574222    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574228    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574231    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574234    2573 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574237    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574240    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574243    2573 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574245    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574249    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574251    2573 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574254    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574261    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574264    2573 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574267    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574272    2573 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574275    2573 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574278    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:31:03.581394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574280    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574283    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574287    2573 flags.go:64] FLAG: --v="2"
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574291    2573 flags.go:64] FLAG: --version="false"
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574297    2573 flags.go:64] FLAG: --vmodule=""
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574301    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574304    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574442    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574445    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574449    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574452    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574455    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574458    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574461    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574464    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574466    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574469    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574472    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574475    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574477    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574480    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574482    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:03.582027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574485    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574487    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574490    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574493    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574496    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574498    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574501    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574504    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574506    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574508    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574511    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574513    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574516    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574518    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574522    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574525    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574527    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574529    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574532    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:03.582585 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574534    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574537    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574540    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574542    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574545    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574547    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574549    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574552    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574555    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574557    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574560    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574563    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574565    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574568    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574570    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574573    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574575    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574578    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574581    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:03.583044 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574584    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574586    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574589    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574591    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574594    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574596    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574598    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574601    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574605    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574607    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574610    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574614    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574617    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574620    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574623    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574627    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574630    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574633    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574636    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:03.583607 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574638    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574641    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574643    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574646    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574649    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574651    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574654    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574657    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574659    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574661    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574664    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574667    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574669    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574672    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574674    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574677    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574679    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574682    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574684    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:03.584062 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574687    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574689    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574694    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574696    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574699    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574701    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574704    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574706    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574709    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574712    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574714    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574716    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574719    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574721    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574724    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.574750    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:31:03.584525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574798    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574803    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574806    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574809    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574812    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574815    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574818    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574820    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574822    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574825    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574828    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574831    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574833    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574836    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574838    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574841    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574843    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574846    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574850    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:03.584920 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574852    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574855    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574857    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574860    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574862    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574865    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574867    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574870    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574873    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574876    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574878    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574881    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574883    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574886    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574888    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574891    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574894    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574896    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574899    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574901    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:03.585434 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574903    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574906    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574909    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574912    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574915    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574917    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574920    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574922    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574925    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574927    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574930    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574933    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574936    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574938    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574941    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574943    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574946    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574949    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574951    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574954    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:03.585921 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574956    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574959    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574961    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574964    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574966    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574969    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574971    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574974    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574976    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574979    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574982    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574984    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574986    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574990    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574993    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574996    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.574999    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575001    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575004    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575006    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:03.586416 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575009    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575011    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575013    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575017    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575020    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575022    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575025    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575027    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575030    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575032    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575035    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575037    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575040    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575042    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575045    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575047    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575049    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575052    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575054    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:03.586886 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575057    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575059    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575062    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575066    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575069    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575072    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575074    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.575080    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.582755    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.582773    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.582791    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.582795    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582876    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582881    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582885    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582887    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582890    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582893    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582895    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:03.587370 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582898    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582901    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582904    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582908    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582912    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582917    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582921    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582926    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582929    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582931    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582934    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582936    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582940    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582943    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582945    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582948    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582951    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582954    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582956    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582959    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:03.587828 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582962    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582964    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582968    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582970    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582973    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582976    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582978    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582981    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582983    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582988    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.582995    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583000    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583004    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583007    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583010    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583014    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583017    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583020    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583023    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:03.588368 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583025    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583028    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583030    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583032    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583035    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583037    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583040    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583042    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583045    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583048    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583051    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583054    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583056    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583060    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583062    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583066    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583071    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583075    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583081    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583085    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:03.588831 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583088    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583091    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583094    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583096    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583099    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583101    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583104    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583106    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583108    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583111    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583114    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583116    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583119    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583121    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583124    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583126    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583128    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583131    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583133    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583136    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:03.589312 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583138    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583141    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583144    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583148    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583171    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583177    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583179    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583182    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583185    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583189    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583191    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583194    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583197    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583199    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583202    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583204    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583207    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583209    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583212    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:03.589781 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583214    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.583250    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583475    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583484    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583490    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583495    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583498    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583500    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583503    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583505    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583510    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583514    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583517    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583520    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583523    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:03.590243 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583525    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583528    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583536    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583539    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583542    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583544    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583547    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583550    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583552    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583555    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583558    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583562    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583567    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583571    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583576    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583578    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583581    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583583    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583586    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:03.590638 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583589    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583591    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583594    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583596    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583599    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583603    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583607    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583610    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583613    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583617    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583619    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583622    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583625    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583627    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583629    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583632    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583643    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583648    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583652    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:03.591076 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583656    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583659    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583662    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583664    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583667    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583670    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583672    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583675    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583677    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583680    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583682    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583684    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583687    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583690    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583692    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583695    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583697    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583700    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583702    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583705    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:03.591537 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583707    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583710    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583713    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583715    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583718    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583723    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583727    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583732    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583735    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583738    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583746    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583749    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583752    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583754    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583757    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583759    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583761    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583764    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583766    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583769    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:03.592513 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583772    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583774    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583777    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583779    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583782    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583785    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583787    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583790    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583792    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583794    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583797    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583801    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583805    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583810    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583814    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:03.593364 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.583839    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583889    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583895    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583899    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583902    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583906    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583910    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583913    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583921    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583924    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583926    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583929    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583931    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583934    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583936    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583939    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583941    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583944    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583947    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:03.593875 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583949    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583952    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583954    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583957    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583960    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583964    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583969    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583973    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583977    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583980    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583983    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583985    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583994    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583996    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.583999    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584003    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584007    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584010    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584013    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:03.594366 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584016    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584019    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584021    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584029    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584032    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584035    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584037    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584047    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584051    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584056    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584060    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584062    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584065    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584068    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584070    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584073    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584075    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584078    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584080    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584083    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:03.594882 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584085    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584087    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584090    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584092    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584095    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584097    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584100    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584102    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584105    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584107    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584110    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584113    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584115    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584117    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584121    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584126    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584142    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584146    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584149    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:03.595525 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584167    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584170    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584173    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584176    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584178    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584181    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584183    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584186    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584188    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584191    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584199    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584204    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584208    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584213    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584217    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584221    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584224    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584226    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584229    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584231    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:03.595993 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584234    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584236    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584239    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584241    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584244    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584247    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584249    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584252    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584254    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:03.584259    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.584327    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.586869    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.587938    2573 server.go:1014] "Starting client certificate rotation"
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.588030    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.588076    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.590182    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:31:03.596533 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.594573    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:31:03.604295 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.604272    2573 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:31:03.608239 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.608225    2573 log.go:25] "Validated CRI v1 image API"
Nov 02 01:31:03.609103 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.609077    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:31:03.612989 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.612971    2573 fs.go:135] Filesystem UUIDs: map[01ab4aaf-b537-4bae-aab8-7a362625c98b:/dev/nvme0n1p3 74d42fd2-0b6b-4e5d-aebc-323443ced95b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 01:31:03.613057 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.612990    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 01:31:03.617398 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.617289    2573 manager.go:217] Machine: {Timestamp:2026-11-02 01:31:03.616799518 +0000 UTC m=+0.281329297 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099174 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2924a6757e535303c5c877aaa3a830 SystemUUID:ec2924a6-757e-5353-03c5-c877aaa3a830 BootID:3c7b2f71-06d1-42ef-a453-3db537f35bf3 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:56:d9:ed:5e:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:56:d9:ed:5e:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:89:ec:b0:b2:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:31:03.617398 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.617398    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:31:03.617500 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.617477    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:31:03.619586 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.619564    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:31:03.619727 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.619588    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-33.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:31:03.619768 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.619729    2573 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:31:03.619768 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.619737    2573 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:31:03.619768 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.619747    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:31:03.619846 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.619768    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:31:03.621045 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.621033    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:31:03.625418 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.625406    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:31:03.625547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.625538    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:31:03.626426 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.626417    2573 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:31:03.626462 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.626434    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:31:03.626462 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.626451    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:31:03.626521 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.626491    2573 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:31:03.626566 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.626538    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:31:03.626940 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.626926    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:31:03.629461 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.629442    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:31:03.636357 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.636341    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:31:03.637293 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.637278    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:31:03.637373 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.637319    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:31:03.637516 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.637503    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:31:03.637577 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.637523    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:31:03.637577 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.637533    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:31:03.637577 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.637541    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:31:03.637577 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.637551    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:31:03.637577 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.637561    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:31:03.638283 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.638273    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:31:03.638342 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.638286    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:31:03.638342 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.638297    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:31:03.638342 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.638306    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:31:03.638342 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.638321    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:31:03.638342 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.638336    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:31:03.640178 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.640150    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:31:03.640237 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.640187    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:31:03.644116 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.644100    2573 server.go:1263] "Started kubelet"
Nov 02 01:31:03.648434 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.648359    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:31:03.648609 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.648517    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:31:03.649294 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.649271    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:31:03.649835 ip-10-0-135-33 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:31:03.653251 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.653231    2573 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:31:03.654336 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.654319    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:31:03.655050 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.655028    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 01:31:03.659997 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.659978    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:31:03.660593 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.660577    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:31:03.660708 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.660691    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-33.us-west-1.compute.internal\" not found"
Nov 02 01:31:03.661376 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.661336    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:31:03.661644 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.661397    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:31:03.661644 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.661413    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:31:03.661644 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.661560    2573 factory.go:55] Registering systemd factory
Nov 02 01:31:03.661644 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.661572    2573 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:31:03.661644 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.661576    2573 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:31:03.661644 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.661579    2573 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:31:03.662014 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.661995    2573 factory.go:153] Registering CRI-O factory
Nov 02 01:31:03.662108 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.662017    2573 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:31:03.662108 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.662067    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:31:03.662108 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.662096    2573 factory.go:103] Registering Raw factory
Nov 02 01:31:03.662244 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.662112    2573 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:31:03.663351 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.663333    2573 manager.go:319] Starting recovery of all containers
Nov 02 01:31:03.673230 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.673216    2573 manager.go:324] Recovery completed
Nov 02 01:31:03.677114 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.677103    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:03.679756 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.679731    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:03.679834 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.679771    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:03.679834 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.679788    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:03.680491 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.680472    2573 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:31:03.680565 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.680491    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:31:03.680565 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.680513    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:31:03.680680 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.680581    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:03.681550 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.681533    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:03.681638 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.681558    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:03.681638 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.681568    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:03.682911 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.682897    2573 policy_none.go:50] "Start"
Nov 02 01:31:03.682977 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.682916    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:31:03.682977 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.682929    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:31:03.684959 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.684947    2573 policy_none.go:44] "Start"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.725445    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.727778    2573 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.727845    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.727854    2573 server.go:87] "Starting device plugin registration server"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.728077    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.728088    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.728183    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.728276    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.728285    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.728819    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.728874    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-33.us-west-1.compute.internal\" not found"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.734542    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.735971    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.735992    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.736008    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:31:03.739143 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.736046    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 01:31:03.789703 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.789659    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-33.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 01:31:03.790833 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.789664    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-33.us-west-1.compute.internal.18e4164b27a969d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-33.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-33.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:31:03.644072402 +0000 UTC m=+0.308602182,LastTimestamp:2026-11-02 01:31:03.644072402 +0000 UTC m=+0.308602182,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-33.us-west-1.compute.internal,}"
Nov 02 01:31:03.828891 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.828840    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:03.829720 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.829705    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:03.829793 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.829736    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:03.829793 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.829750    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:03.829793 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.829773    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:03.836437 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.836421    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"]
Nov 02 01:31:03.836482 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.836477    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:03.837275 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.837252    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:03.837352 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.837286    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:03.837352 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.837302    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:03.838104 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.838086    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-33.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 01:31:03.839086 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.839074    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:03.839796 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.839783    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:03.839855 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.839809    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:03.839855 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.839819    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:03.839855 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.839828    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:03.839940 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.839859    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:03.840481 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.840465    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:03.840540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.840498    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:03.840540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.840512    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:03.841504 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.841490    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:03.841548 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.841515    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:03.842201 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.842186    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:03.842282 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.842213    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:03.842282 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.842228    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:03.867747 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.867722    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:31:03.867864 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.867782    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fnhw5"
Nov 02 01:31:03.881256 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.881231    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-33.us-west-1.compute.internal\" not found" node="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:03.884339 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:03.884323    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-33.us-west-1.compute.internal\" not found" node="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:03.954874 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.954852    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:03.956479 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.956463    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:03.962931 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.962911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:03.963044 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.962936    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:03.963044 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:03.962961    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de43b58832dec4b7f2ea5d9fc15899cf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"de43b58832dec4b7f2ea5d9fc15899cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.004979 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.004952    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:04.055255 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.055228    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:04.055727 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.055700    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.055821 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:04.055741    2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-33.us-west-1.compute.internal\": node \"ip-10-0-135-33.us-west-1.compute.internal\" not found"
Nov 02 01:31:04.061063 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.061050    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.063260 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.063244    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.063316 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.063267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de43b58832dec4b7f2ea5d9fc15899cf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"de43b58832dec4b7f2ea5d9fc15899cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.063316 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.063289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.063395 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.063341    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.063395 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.063346    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.063395 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.063346    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de43b58832dec4b7f2ea5d9fc15899cf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"de43b58832dec4b7f2ea5d9fc15899cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.170598 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.170520    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:31:04.170743 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.170627    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.186865 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.186703    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.188878 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.188857    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:04.226006 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.225985    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:31:04.588496 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.588414    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 01:31:04.589054 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.588631    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:31:04.589054 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.588653    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:31:04.589054 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.588680    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:31:04.589054 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:04.588639    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a86811a309f80445ca4534cfcca7e946-0363337bf0817064.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-33.us-west-1.compute.internal.18e4164b29ca7b17\": read tcp 10.0.135.33:49868->77.113.127.155:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-33.us-west-1.compute.internal.18e4164b29ca7b17  default   4817 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-33.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-33.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-33.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:31:03 +0000 UTC,LastTimestamp:2026-11-02 01:31:03.829754469 +0000 UTC m=+0.494284247,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-33.us-west-1.compute.internal,}"
Nov 02 01:31:04.627710 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.627673    2573 apiserver.go:52] "Watching apiserver"
Nov 02 01:31:04.660754 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.660721    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 01:31:04.872933 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.872835    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:26:05 +0000 UTC" deadline="2028-07-20 02:45:39.959953204 +0000 UTC"
Nov 02 01:31:04.872933 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.872868    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15025h14m35.08708923s"
Nov 02 01:31:04.902089 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.902059    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:31:04.902380 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.902364    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-85l6c","openshift-ovn-kubernetes/ovnkube-node-fxgd2","kube-system/konnectivity-agent-f2jxf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw","openshift-dns/node-resolver-cz4kl","openshift-multus/multus-additional-cni-plugins-z9hpd","openshift-network-diagnostics/network-check-target-hqwck","openshift-network-operator/iptables-alerter-kw282","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal","openshift-multus/multus-mfklz","kube-system/global-pull-secret-syncer-nfs6f","kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-6vlvp","openshift-image-registry/node-ca-9nqbc"]
Nov 02 01:31:04.904576 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.904560    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.906457 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.906432    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:04.906556 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:04.906506    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:04.908321 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.908303    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:04.910315 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.910302    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:04.912205 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.912148    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:04.914228 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.914212    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.916365 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.916346    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:04.916458 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:04.916417    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:04.919406 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.919390    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.921713 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.921697    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:04.923936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.923921    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:04.924004 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:04.923973    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:04.926091 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.926073    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:04.928255 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.928240    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:04.957713 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.957684    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:31:04.957713 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.957684    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:31:04.957713 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.957693    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:31:04.957910 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.957766    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:31:04.957910 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.957805    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vt6x2\""
Nov 02 01:31:04.960925 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.960882    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:31:04.960925 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.960898    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kk5lr\""
Nov 02 01:31:04.961035 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.961004    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:31:04.962051 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.962037    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:31:04.962600 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.962569    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:31:04.962872 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.962857    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:31:04.962920 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.962877    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j52qq\""
Nov 02 01:31:04.962920 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.962899    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:31:04.964503 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.964489    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:31:04.964682 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.964652    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n6fsn\""
Nov 02 01:31:04.964762 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.964752    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:31:04.964884 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.964871    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:31:04.966531 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.966517    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:04.966626 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.966616    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:04.966764 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.966752    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xv9qq\""
Nov 02 01:31:04.967868 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.967849    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-lib-modules\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.967951 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.967884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-kubelet-config\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:04.967951 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.967908    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/445a4994-d804-4369-916f-e9d14143f0c1-hosts-file\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:04.967951 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.967940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-var-lib-kubelet\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.968068 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.967987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-etc-tuned\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.968068 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968034    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cnibin\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:04.968068 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968062    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:04.968219 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25zwf\" (UniqueName: \"kubernetes.io/projected/98ad998e-3ed5-4892-b6d1-f14e516be75a-kube-api-access-25zwf\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.968219 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968130    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-conf\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.968219 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968197    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-netd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.968324 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968224    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58kn4\" (UniqueName: \"kubernetes.io/projected/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-kube-api-access-58kn4\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.968324 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968241    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-system-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.968324 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-conf-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.968324 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968297    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-multus-certs\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.968324 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-device-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:04.968501 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968345    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzsd\" (UniqueName: \"kubernetes.io/projected/a9e45b19-94de-40fc-81c4-960324b99e20-kube-api-access-tpzsd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.968501 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968369    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-etc-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.968501 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968386    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-log-socket\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.968501 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovn-node-metrics-cert\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.968501 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968421    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:04.968501 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968437    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-sys\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.968501 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968459    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.968501 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-script-lib\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.968759 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-sys-fs\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:04.968759 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968551    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-tmp\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.968759 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z69bj\" (UniqueName: \"kubernetes.io/projected/fbde9de9-c649-40ae-a43e-896097fc7dcd-kube-api-access-z69bj\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:04.968759 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-cnibin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.968759 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aba37fc-45ec-4861-b481-f022d9727476-konnectivity-ca\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:04.968759 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968686    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-slash\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.968759 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968709    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.968759 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968725    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:04.968759 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968747    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-socket-dir-parent\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968773    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-k8s-cni-cncf-io\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-hostroot\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968867    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-kubelet\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968932    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbde9de9-c649-40ae-a43e-896097fc7dcd-iptables-alerter-script\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbde9de9-c649-40ae-a43e-896097fc7dcd-host-slash\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.968999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-config\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969028    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e3609b-a3dc-4890-aab1-e4ecd697f718-host\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48e3609b-a3dc-4890-aab1-e4ecd697f718-serviceca\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:04.969077 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-multus\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-env-overrides\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969123    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-system-cni-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969206    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-bin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969224    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aba37fc-45ec-4861-b481-f022d9727476-agent-certs\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969239    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysconfig\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969254    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-host\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969318    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969346    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgnpm\" (UniqueName: \"kubernetes.io/projected/48e3609b-a3dc-4890-aab1-e4ecd697f718-kube-api-access-tgnpm\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw9cx\" (UniqueName: \"kubernetes.io/projected/4d375cf0-b26a-45d5-b6ab-23f752812dd5-kube-api-access-pw9cx\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969420    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-kubernetes\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969443    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-netns\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.969498 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969467    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-systemd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tstcv\" (UniqueName: \"kubernetes.io/projected/adda0d83-929a-4e90-a939-49ba1f7f73b4-kube-api-access-tstcv\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969506    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-systemd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-kubelet\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969535    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-var-lib-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a4994-d804-4369-916f-e9d14143f0c1-tmp-dir\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969564    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-daemon-config\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969591    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-registration-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969628    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-systemd-units\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969651    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-ovn\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969672    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-node-log\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969720    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-run\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-bin\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-dbus\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x422d\" (UniqueName: \"kubernetes.io/projected/445a4994-d804-4369-916f-e9d14143f0c1-kube-api-access-x422d\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:04.970112 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969841    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-cni-binary-copy\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.970561 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969872    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-socket-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:04.970561 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7tt\" (UniqueName: \"kubernetes.io/projected/1b8b7f85-d24b-4ee9-a036-54554ab354ec-kube-api-access-qr7tt\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:04.970561 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969919    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.970561 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-netns\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.970561 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969954    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-etc-kubernetes\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:04.970561 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.969977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-modprobe-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:04.971763 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:04.971746    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xbzw4\""
Nov 02 01:31:05.006098 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.006060    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:31:05.006394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.006280    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:31:05.006394 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.006316    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:31:05.006491 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.006476    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6qvz\""
Nov 02 01:31:05.007952 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.007935    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:05.008522 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.008509    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-87db6\""
Nov 02 01:31:05.008591 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.008580    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:31:05.011818 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.011805    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:31:05.013144 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.013131    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wfswr\""
Nov 02 01:31:05.013225 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.013214    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:31:05.013270 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.013217    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:31:05.013678 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.013665    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:31:05.013827 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.013811    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:31:05.013827 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.013822    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:31:05.013914 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.013849    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:31:05.015136 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.015123    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:05.015374 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.015357    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:31:05.015420 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.015394    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sh6rh"
Nov 02 01:31:05.070588 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbde9de9-c649-40ae-a43e-896097fc7dcd-iptables-alerter-script\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:05.070724 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbde9de9-c649-40ae-a43e-896097fc7dcd-host-slash\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:05.070724 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070613    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-config\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.070724 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070668    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbde9de9-c649-40ae-a43e-896097fc7dcd-host-slash\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:05.070874 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e3609b-a3dc-4890-aab1-e4ecd697f718-host\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:05.070874 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48e3609b-a3dc-4890-aab1-e4ecd697f718-serviceca\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:05.070954 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.070954 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070906    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-multus\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.070954 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070929    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-env-overrides\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.071074 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:05.071074 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070959    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-multus\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.071074 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070979    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-system-cni-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.071074 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.070929    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e3609b-a3dc-4890-aab1-e4ecd697f718-host\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:05.071074 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.071020    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-system-cni-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.071317 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.071096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-bin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.071317 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.071124    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-bin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.071317 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.071185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aba37fc-45ec-4861-b481-f022d9727476-agent-certs\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:05.072040 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072018    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-env-overrides\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.072136 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072059    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysconfig\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.072136 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072065    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbde9de9-c649-40ae-a43e-896097fc7dcd-iptables-alerter-script\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:05.072136 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072021    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysconfig\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.072136 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072058    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:31:05.072136 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072106    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.072136 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-host\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.072136 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072133    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-config\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072130    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072178    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-host\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072175    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072199    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072218    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgnpm\" (UniqueName: \"kubernetes.io/projected/48e3609b-a3dc-4890-aab1-e4ecd697f718-kube-api-access-tgnpm\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072237    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072248    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw9cx\" (UniqueName: \"kubernetes.io/projected/4d375cf0-b26a-45d5-b6ab-23f752812dd5-kube-api-access-pw9cx\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072286    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072302    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-kubernetes\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-netns\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072355    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-netns\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072365    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-kubernetes\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072377    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072383    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-systemd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tstcv\" (UniqueName: \"kubernetes.io/projected/adda0d83-929a-4e90-a939-49ba1f7f73b4-kube-api-access-tstcv\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072415    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48e3609b-a3dc-4890-aab1-e4ecd697f718-serviceca\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:05.072537 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072422    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-systemd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-systemd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072450    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-systemd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072490    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-kubelet\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-var-lib-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a4994-d804-4369-916f-e9d14143f0c1-tmp-dir\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072543    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-kubelet\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-daemon-config\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072558    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-var-lib-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072589    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-registration-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-systemd-units\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072636    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-ovn\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072690    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-ovn\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072699    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-registration-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072700    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-systemd-units\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072723    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-node-log\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072745    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-node-log\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-run\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072790    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a4994-d804-4369-916f-e9d14143f0c1-tmp-dir\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072785    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072807    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-bin\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072832    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-bin\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-dbus\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072861    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-run\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x422d\" (UniqueName: \"kubernetes.io/projected/445a4994-d804-4369-916f-e9d14143f0c1-kube-api-access-x422d\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072899    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-cni-binary-copy\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072903    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-dbus\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072922    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-socket-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072948    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7tt\" (UniqueName: \"kubernetes.io/projected/1b8b7f85-d24b-4ee9-a036-54554ab354ec-kube-api-access-qr7tt\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072973    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.072995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-netns\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-etc-kubernetes\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-modprobe-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073046    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-daemon-config\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-lib-modules\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.073862 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073098    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-etc-kubernetes\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073110    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-kubelet-config\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073125    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-netns\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/445a4994-d804-4369-916f-e9d14143f0c1-hosts-file\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073046    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-socket-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073207    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-modprobe-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073223    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-lib-modules\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-var-lib-kubelet\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073252    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/445a4994-d804-4369-916f-e9d14143f0c1-hosts-file\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073256    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-var-lib-kubelet\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-etc-tuned\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073292    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cnibin\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073305    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-kubelet-config\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073339    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-cni-binary-copy\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073340    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cnibin\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073459    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zwf\" (UniqueName: \"kubernetes.io/projected/98ad998e-3ed5-4892-b6d1-f14e516be75a-kube-api-access-25zwf\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.074696 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.073622    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074026    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-conf\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074082    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-netd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58kn4\" (UniqueName: \"kubernetes.io/projected/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-kube-api-access-58kn4\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074150    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-system-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-conf-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-multus-certs\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074256    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-conf\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074262    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-device-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzsd\" (UniqueName: \"kubernetes.io/projected/a9e45b19-94de-40fc-81c4-960324b99e20-kube-api-access-tpzsd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074324    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-device-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-etc-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-log-socket\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074377    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-netd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovn-node-metrics-cert\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074674    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-system-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074734    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-conf-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074796    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-etc-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.075574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074813    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-multus-certs\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074862    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-log-socket\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.074973    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-sys\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-script-lib\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075119    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-sys\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075128    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-sys-fs\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-tmp\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075209    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-sys-fs\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075237    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69bj\" (UniqueName: \"kubernetes.io/projected/fbde9de9-c649-40ae-a43e-896097fc7dcd-kube-api-access-z69bj\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075259    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-cnibin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075304    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aba37fc-45ec-4861-b481-f022d9727476-konnectivity-ca\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075332    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-slash\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075402    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:05.076302 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-socket-dir-parent\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.077055 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075463    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-k8s-cni-cncf-io\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.078252 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.078226    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-hostroot\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.078337 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.078276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:05.078337 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075773    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-k8s-cni-cncf-io\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.078337 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.078311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.078337 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075838    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075987    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-socket-dir-parent\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.078348    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-kubelet\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.076507    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-etc-tuned\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:05.075928    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.077321    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aba37fc-45ec-4861-b481-f022d9727476-konnectivity-ca\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.076589    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-cnibin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075492    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.075807    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-slash\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.076541    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-script-lib\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.078519 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:05.078507    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:05.078948 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.078599    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-kubelet\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.078948 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:05.078651    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret podName:447b6716-29a3-4000-8df4-022f92de0e23 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:05.57860633 +0000 UTC m=+2.243136119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret") pod "global-pull-secret-syncer-nfs6f" (UID: "447b6716-29a3-4000-8df4-022f92de0e23") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:05.078948 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:05.078696    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:05.578685733 +0000 UTC m=+2.243215501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:05.078948 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.078696    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-hostroot\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.079118 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.078968    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.079118 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.078992    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovn-node-metrics-cert\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.079118 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.079030    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aba37fc-45ec-4861-b481-f022d9727476-agent-certs\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:05.080186 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.080145    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-tmp\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.131015 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:05.130927    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:05.131015 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:05.130960    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:05.131015 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:05.130975    2573 projected.go:196] Error preparing data for projected volume kube-api-access-28lx6 for pod openshift-network-diagnostics/network-check-target-hqwck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:05.131261 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:05.131052    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6 podName:08bd96c5-7bd3-4b79-847c-fd7f7866e231 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:05.631032549 +0000 UTC m=+2.295562330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-28lx6" (UniqueName: "kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6") pod "network-check-target-hqwck" (UID: "08bd96c5-7bd3-4b79-847c-fd7f7866e231") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:05.134030 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.134002    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw9cx\" (UniqueName: \"kubernetes.io/projected/4d375cf0-b26a-45d5-b6ab-23f752812dd5-kube-api-access-pw9cx\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:05.134226 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.134202    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgnpm\" (UniqueName: \"kubernetes.io/projected/48e3609b-a3dc-4890-aab1-e4ecd697f718-kube-api-access-tgnpm\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:05.134635 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.134589    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x422d\" (UniqueName: \"kubernetes.io/projected/445a4994-d804-4369-916f-e9d14143f0c1-kube-api-access-x422d\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:05.134732 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.134670    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7tt\" (UniqueName: \"kubernetes.io/projected/1b8b7f85-d24b-4ee9-a036-54554ab354ec-kube-api-access-qr7tt\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.134732 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.134719    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tstcv\" (UniqueName: \"kubernetes.io/projected/adda0d83-929a-4e90-a939-49ba1f7f73b4-kube-api-access-tstcv\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.134926 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.134909    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zwf\" (UniqueName: \"kubernetes.io/projected/98ad998e-3ed5-4892-b6d1-f14e516be75a-kube-api-access-25zwf\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.135987 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.135963    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58kn4\" (UniqueName: \"kubernetes.io/projected/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-kube-api-access-58kn4\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.136056 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.135982    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzsd\" (UniqueName: \"kubernetes.io/projected/a9e45b19-94de-40fc-81c4-960324b99e20-kube-api-access-tpzsd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.137138 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.137120    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69bj\" (UniqueName: \"kubernetes.io/projected/fbde9de9-c649-40ae-a43e-896097fc7dcd-kube-api-access-z69bj\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:05.224709 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.224679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mfklz"
Nov 02 01:31:05.246390 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.246365    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:05.267979 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.267957    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:05.285682 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.285658    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:05.305399 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.305366    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:05.337511 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.337481    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:05.357117 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.357090    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:05.365655 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.365634    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:05.369957 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.369943    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:05.428493 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:05.428464    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fc15b85861deeb88c7e9a1573ac9e6.slice/crio-6cb6b5398a193c429473f822e2aca3b4a96addbdabbf64994373462fff87c691 WatchSource:0}: Error finding container 6cb6b5398a193c429473f822e2aca3b4a96addbdabbf64994373462fff87c691: Status 404 returned error can't find the container with id 6cb6b5398a193c429473f822e2aca3b4a96addbdabbf64994373462fff87c691
Nov 02 01:31:05.428901 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:05.428880    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde43b58832dec4b7f2ea5d9fc15899cf.slice/crio-3d775d12f8252fb7e43a677dc50a3dfcc8672c5f19f26fcb464538f80298c5ef WatchSource:0}: Error finding container 3d775d12f8252fb7e43a677dc50a3dfcc8672c5f19f26fcb464538f80298c5ef: Status 404 returned error can't find the container with id 3d775d12f8252fb7e43a677dc50a3dfcc8672c5f19f26fcb464538f80298c5ef
Nov 02 01:31:05.434255 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:05.434241    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:31:07.450308 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.450283    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:07.482273 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.482242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:07.482273 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.482280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:07.482472 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:07.482391    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:07.482472 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:07.482393    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:07.482472 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:07.482441    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:08.48242922 +0000 UTC m=+3.248058469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:07.482472 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:07.482466    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret podName:447b6716-29a3-4000-8df4-022f92de0e23 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:08.482460311 +0000 UTC m=+3.248089557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret") pod "global-pull-secret-syncer-nfs6f" (UID: "447b6716-29a3-4000-8df4-022f92de0e23") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:07.502252 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.502228    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:07.583286 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.583200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:07.583454 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:07.583356    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:07.583454 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:07.583375    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:07.583454 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:07.583386    2573 projected.go:196] Error preparing data for projected volume kube-api-access-28lx6 for pod openshift-network-diagnostics/network-check-target-hqwck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:07.583454 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:07.583446    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6 podName:08bd96c5-7bd3-4b79-847c-fd7f7866e231 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:08.58342999 +0000 UTC m=+3.349059241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-28lx6" (UniqueName: "kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6") pod "network-check-target-hqwck" (UID: "08bd96c5-7bd3-4b79-847c-fd7f7866e231") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:07.640237 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.640175    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"de43b58832dec4b7f2ea5d9fc15899cf","Type":"ContainerStarted","Data":"3d775d12f8252fb7e43a677dc50a3dfcc8672c5f19f26fcb464538f80298c5ef"}
Nov 02 01:31:07.641507 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.641482    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"95fc15b85861deeb88c7e9a1573ac9e6","Type":"ContainerStarted","Data":"6cb6b5398a193c429473f822e2aca3b4a96addbdabbf64994373462fff87c691"}
Nov 02 01:31:07.792012 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.791981    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:07.917642 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.917569    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:26:06 +0000 UTC" deadline="2028-04-30 11:06:31.519499981 +0000 UTC"
Nov 02 01:31:07.917642 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:07.917593    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13089h35m23.601909575s"
Nov 02 01:31:07.924245 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:07.924222    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ad998e_3ed5_4892_b6d1_f14e516be75a.slice/crio-8cb6fff0d1293c6a6ae781628682e876f986b914f1c4dda9fd3ce8c5643c82a1 WatchSource:0}: Error finding container 8cb6fff0d1293c6a6ae781628682e876f986b914f1c4dda9fd3ce8c5643c82a1: Status 404 returned error can't find the container with id 8cb6fff0d1293c6a6ae781628682e876f986b914f1c4dda9fd3ce8c5643c82a1
Nov 02 01:31:07.965608 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:07.965577    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aba37fc_45ec_4861_b481_f022d9727476.slice/crio-3319a73e68a40d31273879a6d1d180caa0d8bf00badba0da6ef585a818c2508a WatchSource:0}: Error finding container 3319a73e68a40d31273879a6d1d180caa0d8bf00badba0da6ef585a818c2508a: Status 404 returned error can't find the container with id 3319a73e68a40d31273879a6d1d180caa0d8bf00badba0da6ef585a818c2508a
Nov 02 01:31:08.017016 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:08.016989    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadda0d83_929a_4e90_a939_49ba1f7f73b4.slice/crio-1eb0faf71a80468585c038798d61123dfce42682f18aade978e135ca9d2535ed WatchSource:0}: Error finding container 1eb0faf71a80468585c038798d61123dfce42682f18aade978e135ca9d2535ed: Status 404 returned error can't find the container with id 1eb0faf71a80468585c038798d61123dfce42682f18aade978e135ca9d2535ed
Nov 02 01:31:08.040409 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:08.040370    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e3609b_a3dc_4890_aab1_e4ecd697f718.slice/crio-51705617cf3a2ecf3259819f8edf49e6d6572dff388b38cfcac85423b2b9678c WatchSource:0}: Error finding container 51705617cf3a2ecf3259819f8edf49e6d6572dff388b38cfcac85423b2b9678c: Status 404 returned error can't find the container with id 51705617cf3a2ecf3259819f8edf49e6d6572dff388b38cfcac85423b2b9678c
Nov 02 01:31:08.041002 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:08.040981    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e45b19_94de_40fc_81c4_960324b99e20.slice/crio-0533924d1a6dd23ac07018ae181af73512265727e91c3c8511f9e1633e166bd4 WatchSource:0}: Error finding container 0533924d1a6dd23ac07018ae181af73512265727e91c3c8511f9e1633e166bd4: Status 404 returned error can't find the container with id 0533924d1a6dd23ac07018ae181af73512265727e91c3c8511f9e1633e166bd4
Nov 02 01:31:08.063357 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:08.063326    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df2f3d5_e8eb_48e4_b196_2c08c1fd28ce.slice/crio-790fcf472d67dced95cbb5e648ef4c6d0d66e731b42699588dd0c0ebf1975410 WatchSource:0}: Error finding container 790fcf472d67dced95cbb5e648ef4c6d0d66e731b42699588dd0c0ebf1975410: Status 404 returned error can't find the container with id 790fcf472d67dced95cbb5e648ef4c6d0d66e731b42699588dd0c0ebf1975410
Nov 02 01:31:08.100332 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:08.100298    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8b7f85_d24b_4ee9_a036_54554ab354ec.slice/crio-934ff7aa8060dc5cf7622f64ca2125527dc3448dac4ff74059aaa8838df4694f WatchSource:0}: Error finding container 934ff7aa8060dc5cf7622f64ca2125527dc3448dac4ff74059aaa8838df4694f: Status 404 returned error can't find the container with id 934ff7aa8060dc5cf7622f64ca2125527dc3448dac4ff74059aaa8838df4694f
Nov 02 01:31:08.101896 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:08.101868    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbde9de9_c649_40ae_a43e_896097fc7dcd.slice/crio-5b08196084d613a5f329f0ff23b4e4ba494a840c36dc11d86da8844fe5f9d720 WatchSource:0}: Error finding container 5b08196084d613a5f329f0ff23b4e4ba494a840c36dc11d86da8844fe5f9d720: Status 404 returned error can't find the container with id 5b08196084d613a5f329f0ff23b4e4ba494a840c36dc11d86da8844fe5f9d720
Nov 02 01:31:08.118115 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:08.118088    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445a4994_d804_4369_916f_e9d14143f0c1.slice/crio-7ac78d5863a5ee181baa4aa429370961546a266d1891d38161a98b6864b1a6fb WatchSource:0}: Error finding container 7ac78d5863a5ee181baa4aa429370961546a266d1891d38161a98b6864b1a6fb: Status 404 returned error can't find the container with id 7ac78d5863a5ee181baa4aa429370961546a266d1891d38161a98b6864b1a6fb
Nov 02 01:31:08.491015 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.490979    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:08.491015 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.491025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:08.491281 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.491172    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:08.491281 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.491172    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:08.491281 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.491233    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:10.491215784 +0000 UTC m=+5.256845048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:08.491281 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.491251    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret podName:447b6716-29a3-4000-8df4-022f92de0e23 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:10.491241862 +0000 UTC m=+5.256871108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret") pod "global-pull-secret-syncer-nfs6f" (UID: "447b6716-29a3-4000-8df4-022f92de0e23") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:08.591902 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.591853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:08.592437 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.591995    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:08.592437 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.592012    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:08.592437 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.592021    2573 projected.go:196] Error preparing data for projected volume kube-api-access-28lx6 for pod openshift-network-diagnostics/network-check-target-hqwck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:08.592437 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.592088    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6 podName:08bd96c5-7bd3-4b79-847c-fd7f7866e231 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:10.592073657 +0000 UTC m=+5.357702923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-28lx6" (UniqueName: "kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6") pod "network-check-target-hqwck" (UID: "08bd96c5-7bd3-4b79-847c-fd7f7866e231") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:08.635317 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.635287    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:08.635482 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.635451    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:08.635569 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.635548    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:08.635647 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.635628    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:08.635742 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.635709    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:08.635878 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:08.635849    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:08.646117 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.645573    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cz4kl" event={"ID":"445a4994-d804-4369-916f-e9d14143f0c1","Type":"ContainerStarted","Data":"7ac78d5863a5ee181baa4aa429370961546a266d1891d38161a98b6864b1a6fb"}
Nov 02 01:31:08.650388 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.650358    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerStarted","Data":"790fcf472d67dced95cbb5e648ef4c6d0d66e731b42699588dd0c0ebf1975410"}
Nov 02 01:31:08.651829 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.651805    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6vlvp" event={"ID":"a9e45b19-94de-40fc-81c4-960324b99e20","Type":"ContainerStarted","Data":"0533924d1a6dd23ac07018ae181af73512265727e91c3c8511f9e1633e166bd4"}
Nov 02 01:31:08.653291 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.653253    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f2jxf" event={"ID":"2aba37fc-45ec-4861-b481-f022d9727476","Type":"ContainerStarted","Data":"3319a73e68a40d31273879a6d1d180caa0d8bf00badba0da6ef585a818c2508a"}
Nov 02 01:31:08.654553 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.654529    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mfklz" event={"ID":"98ad998e-3ed5-4892-b6d1-f14e516be75a","Type":"ContainerStarted","Data":"8cb6fff0d1293c6a6ae781628682e876f986b914f1c4dda9fd3ce8c5643c82a1"}
Nov 02 01:31:08.656196 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.656172    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kw282" event={"ID":"fbde9de9-c649-40ae-a43e-896097fc7dcd","Type":"ContainerStarted","Data":"5b08196084d613a5f329f0ff23b4e4ba494a840c36dc11d86da8844fe5f9d720"}
Nov 02 01:31:08.657790 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.657767    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9hpd" event={"ID":"1b8b7f85-d24b-4ee9-a036-54554ab354ec","Type":"ContainerStarted","Data":"934ff7aa8060dc5cf7622f64ca2125527dc3448dac4ff74059aaa8838df4694f"}
Nov 02 01:31:08.659259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.659231    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9nqbc" event={"ID":"48e3609b-a3dc-4890-aab1-e4ecd697f718","Type":"ContainerStarted","Data":"51705617cf3a2ecf3259819f8edf49e6d6572dff388b38cfcac85423b2b9678c"}
Nov 02 01:31:08.660749 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.660728    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw" event={"ID":"adda0d83-929a-4e90-a939-49ba1f7f73b4","Type":"ContainerStarted","Data":"1eb0faf71a80468585c038798d61123dfce42682f18aade978e135ca9d2535ed"}
Nov 02 01:31:08.918418 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.918281    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:26:06 +0000 UTC" deadline="2028-04-29 21:22:04.195310709 +0000 UTC"
Nov 02 01:31:08.918418 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:08.918373    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13075h50m55.276941436s"
Nov 02 01:31:10.506791 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:10.506756    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:10.507410 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:10.506806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:10.507410 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.506965    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:10.507410 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.507024    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:14.507007442 +0000 UTC m=+9.272636702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:10.507410 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.507104    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:10.507410 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.507139    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret podName:447b6716-29a3-4000-8df4-022f92de0e23 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:14.507128796 +0000 UTC m=+9.272758042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret") pod "global-pull-secret-syncer-nfs6f" (UID: "447b6716-29a3-4000-8df4-022f92de0e23") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:10.607469 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:10.607413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:10.607656 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.607604    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:10.607656 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.607623    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:10.607656 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.607639    2573 projected.go:196] Error preparing data for projected volume kube-api-access-28lx6 for pod openshift-network-diagnostics/network-check-target-hqwck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:10.607815 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.607697    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6 podName:08bd96c5-7bd3-4b79-847c-fd7f7866e231 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:14.607679366 +0000 UTC m=+9.373308615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-28lx6" (UniqueName: "kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6") pod "network-check-target-hqwck" (UID: "08bd96c5-7bd3-4b79-847c-fd7f7866e231") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:10.636119 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:10.636087    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:10.636304 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.636223    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:10.636304 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:10.636291    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:10.636484 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.636361    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:10.636484 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:10.636405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:10.636484 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:10.636469    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:12.636018 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.635936    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:12.636471 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:12.636086    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:12.636471 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.636165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:12.636471 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:12.636243    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:12.636471 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.636302    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:12.636471 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:12.636374    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:12.697492 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.697456    2573 generic.go:356] "Generic (PLEG): container finished" podID="95fc15b85861deeb88c7e9a1573ac9e6" containerID="a90397d2c85a500c6b08b085d31066513f68c9457082b69ca36ed98a88caea6c" exitCode=0
Nov 02 01:31:12.697666 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.697544    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"95fc15b85861deeb88c7e9a1573ac9e6","Type":"ContainerDied","Data":"a90397d2c85a500c6b08b085d31066513f68c9457082b69ca36ed98a88caea6c"}
Nov 02 01:31:12.712709 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.712675    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="6a05dfd8de359cbd78f44720b357d757838bff8aa659dea85d81676201d44735" exitCode=0
Nov 02 01:31:12.712840 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.712772    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9hpd" event={"ID":"1b8b7f85-d24b-4ee9-a036-54554ab354ec","Type":"ContainerDied","Data":"6a05dfd8de359cbd78f44720b357d757838bff8aa659dea85d81676201d44735"}
Nov 02 01:31:12.722824 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.722795    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f2jxf" event={"ID":"2aba37fc-45ec-4861-b481-f022d9727476","Type":"ContainerStarted","Data":"2bd217798cc5c67ad8af971fe3f0baa39e6f285d05b94ecab251e28efb02c7e4"}
Nov 02 01:31:12.727805 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.727777    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"de43b58832dec4b7f2ea5d9fc15899cf","Type":"ContainerStarted","Data":"a29aa03df596f6bee6ece859190af9a79b572c06c69b87adbe381bddeef7911b"}
Nov 02 01:31:12.931497 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:12.931178    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal" podStartSLOduration=6.931158705 podStartE2EDuration="6.931158705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:12.93010345 +0000 UTC m=+7.695732746" watchObservedRunningTime="2026-11-02 01:31:12.931158705 +0000 UTC m=+7.696787974"
Nov 02 01:31:13.168519 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:13.167312    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f2jxf" podStartSLOduration=3.747165753 podStartE2EDuration="8.167290951s" totalImagesPullingTime="4.420125198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:13.046252195 +0000 UTC m=+7.811881465" watchObservedRunningTime="2026-11-02 01:31:13.167290951 +0000 UTC m=+7.932920216"
Nov 02 01:31:13.751726 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:13.751696    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"95fc15b85861deeb88c7e9a1573ac9e6","Type":"ContainerStarted","Data":"a8fc6fd72aa58d7277a892a5f790b2d2c7da8b4f3326f6f66e757042303ee482"}
Nov 02 01:31:13.826848 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:13.826817    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:13.827455 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:13.827426    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:13.984246 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:13.983830    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal" podStartSLOduration=7.9838078679999995 podStartE2EDuration="7.983807868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:13.867422521 +0000 UTC m=+8.633051789" watchObservedRunningTime="2026-11-02 01:31:13.983807868 +0000 UTC m=+8.749437145"
Nov 02 01:31:14.547035 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:14.546548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:14.547035 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:14.546599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:14.547035 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.546747    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:14.547035 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.546817    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:22.546796828 +0000 UTC m=+17.312426077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:14.547035 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.546888    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:14.547035 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.546924    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret podName:447b6716-29a3-4000-8df4-022f92de0e23 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:22.546913866 +0000 UTC m=+17.312543126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret") pod "global-pull-secret-syncer-nfs6f" (UID: "447b6716-29a3-4000-8df4-022f92de0e23") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:14.636138 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:14.635792    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:14.636138 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.635925    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:14.636138 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:14.636006    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:14.636777 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.636542    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:14.636777 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:14.636629    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:14.636777 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.636709    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:14.647778 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:14.647492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:14.647778 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.647662    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:14.647778 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.647683    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:14.647778 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.647694    2573 projected.go:196] Error preparing data for projected volume kube-api-access-28lx6 for pod openshift-network-diagnostics/network-check-target-hqwck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:14.647778 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:14.647745    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6 podName:08bd96c5-7bd3-4b79-847c-fd7f7866e231 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:22.647726802 +0000 UTC m=+17.413356063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-28lx6" (UniqueName: "kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6") pod "network-check-target-hqwck" (UID: "08bd96c5-7bd3-4b79-847c-fd7f7866e231") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:14.756863 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:14.756196    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9nqbc" event={"ID":"48e3609b-a3dc-4890-aab1-e4ecd697f718","Type":"ContainerStarted","Data":"d548843e460dd54ad71639bbad7ef46077b19e215f428aceb67603e73bce1727"}
Nov 02 01:31:14.760574 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:14.760543    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw" event={"ID":"adda0d83-929a-4e90-a939-49ba1f7f73b4","Type":"ContainerStarted","Data":"3bfdf740e671a82b6d779dcf82d0921f63e35e6ff40f2be51cd04617a45d751a"}
Nov 02 01:31:15.763910 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:15.763882    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:31:16.475451 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.475416    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:31:16.633484 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.632665    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:31:16.475445928Z","UUID":"27c21ba1-02ac-44c0-b5a2-87ed55428e78","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:31:16.635847 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.635396    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:16.635847 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:16.635518    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:16.635847 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.635580    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:16.635847 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:16.635660    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:16.635847 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.635710    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:16.635847 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:16.635779    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:16.637173 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.636556    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:31:16.637173 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.636581    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:31:16.768743 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.768028    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mfklz" event={"ID":"98ad998e-3ed5-4892-b6d1-f14e516be75a","Type":"ContainerStarted","Data":"3463aeb2a8001b4d7fc948a85d64d737359ba7d8faa2898a05cace3ed8334a4f"}
Nov 02 01:31:16.771312 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.771279    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw" event={"ID":"adda0d83-929a-4e90-a939-49ba1f7f73b4","Type":"ContainerStarted","Data":"c18d823010cb4a04ad94eaf38f0757f4d4467bfb204a552b5c53c97cb8c42bb1"}
Nov 02 01:31:16.889961 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.889455    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mfklz" podStartSLOduration=3.625093232 podStartE2EDuration="11.889437433s" totalImagesPullingTime="8.264344201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:16.88878849 +0000 UTC m=+11.654417756" watchObservedRunningTime="2026-11-02 01:31:16.889437433 +0000 UTC m=+11.655066702"
Nov 02 01:31:16.889961 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:16.889757    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9nqbc" podStartSLOduration=6.219632503 podStartE2EDuration="11.889750832s" totalImagesPullingTime="5.670118329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:14.873179616 +0000 UTC m=+9.638808885" watchObservedRunningTime="2026-11-02 01:31:16.889750832 +0000 UTC m=+11.655380102"
Nov 02 01:31:17.775836 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:17.775797    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="66887026d9c3a83aab9d9df27453f3bbab0320828733379070ce9a6de573c109" exitCode=0
Nov 02 01:31:17.776559 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:17.776527    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9hpd" event={"ID":"1b8b7f85-d24b-4ee9-a036-54554ab354ec","Type":"ContainerDied","Data":"66887026d9c3a83aab9d9df27453f3bbab0320828733379070ce9a6de573c109"}
Nov 02 01:31:17.779238 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:17.779209    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cz4kl" event={"ID":"445a4994-d804-4369-916f-e9d14143f0c1","Type":"ContainerStarted","Data":"2ba2b2171428c8a7a36dd9a07bc1486c7c724abffcacef6282d9854f71ef0c35"}
Nov 02 01:31:18.635798 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:18.635638    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:18.635882 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:18.635674    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:18.635932 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:18.635899    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:18.636005 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:18.635980    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:18.636074 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:18.635713    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:18.636134 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:18.636106    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:18.783303 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:18.783261    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kw282" event={"ID":"fbde9de9-c649-40ae-a43e-896097fc7dcd","Type":"ContainerStarted","Data":"81a05f5bff00417556754e8a5287b245cb47e02fd2e0fa6026f64c0be0cabdd1"}
Nov 02 01:31:18.785875 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:18.785777    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw" event={"ID":"adda0d83-929a-4e90-a939-49ba1f7f73b4","Type":"ContainerStarted","Data":"0b44208879fedcc4d39b79843ed54ad45d1ef206ffff9debed8a11a6ef89eb2e"}
Nov 02 01:31:18.787996 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:18.787964    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6vlvp" event={"ID":"a9e45b19-94de-40fc-81c4-960324b99e20","Type":"ContainerStarted","Data":"221c413bf045831d72976db7a7bcd2575fddfab322bf00d571f13b731a4bfee9"}
Nov 02 01:31:18.901101 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:18.901004    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cz4kl" podStartSLOduration=4.149500051 podStartE2EDuration="12.90099113s" totalImagesPullingTime="8.751491079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:18.014197537 +0000 UTC m=+12.779826804" watchObservedRunningTime="2026-11-02 01:31:18.90099113 +0000 UTC m=+13.666620398"
Nov 02 01:31:18.901234 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:18.901184    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kw282" podStartSLOduration=4.379645745 podStartE2EDuration="12.901179783s" totalImagesPullingTime="8.521534038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:18.900285275 +0000 UTC m=+13.665914542" watchObservedRunningTime="2026-11-02 01:31:18.901179783 +0000 UTC m=+13.666809051"
Nov 02 01:31:19.020281 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:19.020231    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6vlvp" podStartSLOduration=2.441799957 podStartE2EDuration="13.020211917s" totalImagesPullingTime="10.57841196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:19.019571279 +0000 UTC m=+13.785200547" watchObservedRunningTime="2026-11-02 01:31:19.020211917 +0000 UTC m=+13.785841187"
Nov 02 01:31:19.791295 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:19.791259    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="a0f3aa432d8427f3add5ed43997709d0fb1c6e1124082122f7e2595a4a35bd66" exitCode=0
Nov 02 01:31:19.791833 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:19.791360    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9hpd" event={"ID":"1b8b7f85-d24b-4ee9-a036-54554ab354ec","Type":"ContainerDied","Data":"a0f3aa432d8427f3add5ed43997709d0fb1c6e1124082122f7e2595a4a35bd66"}
Nov 02 01:31:19.914148 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:19.914099    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw" podStartSLOduration=5.22747652 podStartE2EDuration="14.914085719s" totalImagesPullingTime="9.686609199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:19.139997624 +0000 UTC m=+13.905626893" watchObservedRunningTime="2026-11-02 01:31:19.914085719 +0000 UTC m=+14.679715017"
Nov 02 01:31:20.635858 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:20.635710    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:20.635858 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:20.635812    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:20.635858 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:20.635833    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:20.636157 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:20.635954    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:20.636157 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:20.636071    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:20.636157 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:20.636138    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:21.799011 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:21.798974    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="45c6df80488d4d59bf4dfae174ecc73f73786d09b19560e449772ceec546ed07" exitCode=0
Nov 02 01:31:21.799779 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:21.799036    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9hpd" event={"ID":"1b8b7f85-d24b-4ee9-a036-54554ab354ec","Type":"ContainerDied","Data":"45c6df80488d4d59bf4dfae174ecc73f73786d09b19560e449772ceec546ed07"}
Nov 02 01:31:22.611891 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:22.611849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:22.612101 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:22.611898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:22.612101 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.611979    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:22.612101 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.611993    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:22.612101 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.612070    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:38.612035366 +0000 UTC m=+33.377664616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:22.612101 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.612091    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret podName:447b6716-29a3-4000-8df4-022f92de0e23 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:38.612080258 +0000 UTC m=+33.377709507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret") pod "global-pull-secret-syncer-nfs6f" (UID: "447b6716-29a3-4000-8df4-022f92de0e23") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:22.636156 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:22.636128    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:22.636156 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:22.636158    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:22.636366 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:22.636129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:22.636366 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.636259    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:22.636366 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.636326    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:22.636490 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.636412    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:22.712846 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:22.712805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:22.713020 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.712961    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:22.713020 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.712978    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:22.713020 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.712987    2573 projected.go:196] Error preparing data for projected volume kube-api-access-28lx6 for pod openshift-network-diagnostics/network-check-target-hqwck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:22.713171 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:22.713042    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6 podName:08bd96c5-7bd3-4b79-847c-fd7f7866e231 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:38.713023636 +0000 UTC m=+33.478652891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-28lx6" (UniqueName: "kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6") pod "network-check-target-hqwck" (UID: "08bd96c5-7bd3-4b79-847c-fd7f7866e231") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:23.808198 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:23.807909    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:31:23.808662 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:23.808576    2573 generic.go:356] "Generic (PLEG): container finished" podID="5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce" containerID="16d22453786eef61b8dd76c46746af591321cb86d3e4c77e9cbea7deb8251035" exitCode=1
Nov 02 01:31:23.808662 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:23.808617    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerStarted","Data":"f114168cc198f672a7b8b6e1f98e96c794d78655fa13cc69649eb801e821ae35"}
Nov 02 01:31:23.808776 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:23.808692    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerStarted","Data":"dd4c9f8a2b2484572e0925a037d9595847b604790b225ee8440e479162867f06"}
Nov 02 01:31:23.808776 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:23.808740    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerStarted","Data":"68e7d853e019d8f573e1b82cbe5f8a0c255f7574e25272ca853611e0ed8f7555"}
Nov 02 01:31:23.808883 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:23.808789    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerStarted","Data":"b9f7e7d7bfd9c0cc9e03ec52cfc5c864c2f1c9101f6bf2fef54d41ce5b0b1e61"}
Nov 02 01:31:23.808883 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:23.808837    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerDied","Data":"16d22453786eef61b8dd76c46746af591321cb86d3e4c77e9cbea7deb8251035"}
Nov 02 01:31:23.808966 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:23.808885    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerStarted","Data":"b094e0897a1a1c205ce362a05e0ce462af7eb74f37466f981a73e4e07248d80e"}
Nov 02 01:31:24.143320 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:24.143153    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:24.143463 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:24.143422    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:31:24.143798 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:24.143780    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:24.635275 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:24.635236    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:24.635275 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:24.635264    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:24.635456 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:24.635386    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:24.635456 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:24.635422    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:24.635535 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:24.635499    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:24.635571 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:24.635556    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:24.813679 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:24.813649    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="ba24fd59729e757849291b4423534833b33b1b3f04884f8f6d2a7a3ee46501e6" exitCode=0
Nov 02 01:31:24.814098 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:24.813745    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9hpd" event={"ID":"1b8b7f85-d24b-4ee9-a036-54554ab354ec","Type":"ContainerDied","Data":"ba24fd59729e757849291b4423534833b33b1b3f04884f8f6d2a7a3ee46501e6"}
Nov 02 01:31:25.819178 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:25.819128    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="e5dbe11c75f577477a8e15839e430185b62110d018ef410ae27d882b754cd937" exitCode=0
Nov 02 01:31:25.819663 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:25.819256    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9hpd" event={"ID":"1b8b7f85-d24b-4ee9-a036-54554ab354ec","Type":"ContainerDied","Data":"e5dbe11c75f577477a8e15839e430185b62110d018ef410ae27d882b754cd937"}
Nov 02 01:31:25.825154 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:25.825133    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:31:25.825658 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:25.825595    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerStarted","Data":"20f7de367cc19db46bff9d46ddc6d246e192a8c5b40e9162c793522eb395ec49"}
Nov 02 01:31:26.635392 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:26.635354    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:26.635392 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:26.635393    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:26.635584 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:26.635400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:26.635584 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:26.635499    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:26.635672 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:26.635610    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:26.635714 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:26.635695    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:26.830476 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:26.830448    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9hpd" event={"ID":"1b8b7f85-d24b-4ee9-a036-54554ab354ec","Type":"ContainerStarted","Data":"3bcc520bbc95fff35ea2e88b71507d2c165ecc1e081f4ac5403d38862b035f28"}
Nov 02 01:31:26.953628 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:26.953536    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z9hpd" podStartSLOduration=8.121862379 podStartE2EDuration="20.953521898s" totalImagesPullingTime="12.831659519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:26.953110937 +0000 UTC m=+21.718740210" watchObservedRunningTime="2026-11-02 01:31:26.953521898 +0000 UTC m=+21.719151167"
Nov 02 01:31:27.836547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:27.836523    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:31:28.635109 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:28.635082    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:28.635109 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:28.635097    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:28.635538 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:28.635202    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:28.635538 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:28.635219    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:28.638567 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:28.638537    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:28.638567 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:28.638557    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:28.842929 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:28.842901    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:31:28.843380 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:28.843289    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerStarted","Data":"c2bd7d1e9a92eca3cfb9e676ef0ad3a2d44d93b79abad3feaa105fa0a5d00a0e"}
Nov 02 01:31:28.843563 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:28.843543    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:28.843624 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:28.843604    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:28.843705 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:28.843690    2573 scope.go:122] "RemoveContainer" containerID="16d22453786eef61b8dd76c46746af591321cb86d3e4c77e9cbea7deb8251035"
Nov 02 01:31:28.861030 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:28.860879    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:29.850177 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:29.850151    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:31:29.850629 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:29.850503    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" event={"ID":"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce","Type":"ContainerStarted","Data":"1ef9aac94d74e63e08f8f1018206652d76dcd296e17eaade3e9c3349f21deaf2"}
Nov 02 01:31:29.850921 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:29.850905    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:29.867311 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:29.867280    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:30.030457 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:30.030386    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2" podStartSLOduration=8.959545424 podStartE2EDuration="24.030370573s" totalImagesPullingTime="15.070825149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:30.029858914 +0000 UTC m=+24.795488182" watchObservedRunningTime="2026-11-02 01:31:30.030370573 +0000 UTC m=+24.795999868"
Nov 02 01:31:30.571150 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:30.571102    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nfs6f"]
Nov 02 01:31:30.571321 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:30.571287    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:30.571460 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:30.571434    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:30.575505 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:30.575247    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hqwck"]
Nov 02 01:31:30.575505 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:30.575372    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:30.575505 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:30.575471    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:30.575751 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:30.575732    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-85l6c"]
Nov 02 01:31:30.575829 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:30.575818    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:30.575915 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:30.575899    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:31.635732 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:31.635699    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:31.636126 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:31.635844    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:32.635154 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:32.635112    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:32.635349 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:32.635113    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:32.635349 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:32.635221    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:32.635349 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:32.635297    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:33.635552 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:33.635513    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:33.636021 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:33.635675    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:34.635865 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:34.635785    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:34.636251 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:34.635785    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:34.636251 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:34.635895    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqwck" podUID="08bd96c5-7bd3-4b79-847c-fd7f7866e231"
Nov 02 01:31:34.636251 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:34.635970    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfs6f" podUID="447b6716-29a3-4000-8df4-022f92de0e23"
Nov 02 01:31:35.636789 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:35.636708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:35.637230 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:35.636833    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85l6c" podUID="4d375cf0-b26a-45d5-b6ab-23f752812dd5"
Nov 02 01:31:35.948797 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:35.948766    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeReady"
Nov 02 01:31:35.948982 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:35.948917    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 01:31:36.107285 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.107250    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-57lbv"]
Nov 02 01:31:36.117144 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.117118    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5tvxk"]
Nov 02 01:31:36.117299 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.117253    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-57lbv"
Nov 02 01:31:36.127630 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.127442    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xkcjd"]
Nov 02 01:31:36.127739 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.127548    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5tvxk"
Nov 02 01:31:36.141543 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.141498    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"]
Nov 02 01:31:36.141621 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.141612    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xkcjd"
Nov 02 01:31:36.149871 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.149852    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"]
Nov 02 01:31:36.149956 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.149945    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:36.165895 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.165872    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bv7fm"]
Nov 02 01:31:36.166001 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.165973    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"
Nov 02 01:31:36.177161 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.177124    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.177273 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.177216    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.179641 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.179618    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"]
Nov 02 01:31:36.180917 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.180901    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.195834 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.195817    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7948cc5b95-b87qg"]
Nov 02 01:31:36.195952 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.195935    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.202619 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.202600    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 01:31:36.202813 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.202802    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 01:31:36.206109 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.206092    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7xqs\" (UniqueName: \"kubernetes.io/projected/444113a1-c287-4a62-98ba-7b88fe123732-kube-api-access-x7xqs\") pod \"kas-connection-checker-685895c7c9-5tvxk\" (UID: \"444113a1-c287-4a62-98ba-7b88fe123732\") " pod="kube-system/kas-connection-checker-685895c7c9-5tvxk"
Nov 02 01:31:36.206164 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.206132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8kn5\" (UniqueName: \"kubernetes.io/projected/230459c8-baa4-47cc-9cc2-73f0535fb650-kube-api-access-r8kn5\") pod \"kas-connection-checker-685895c7c9-xkcjd\" (UID: \"230459c8-baa4-47cc-9cc2-73f0535fb650\") " pod="kube-system/kas-connection-checker-685895c7c9-xkcjd"
Nov 02 01:31:36.206164 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.206154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt47v\" (UniqueName: \"kubernetes.io/projected/f74b04ce-df4a-46d3-9eb5-52464d2b5e50-kube-api-access-jt47v\") pod \"kas-connection-checker-685895c7c9-57lbv\" (UID: \"f74b04ce-df4a-46d3-9eb5-52464d2b5e50\") " pod="kube-system/kas-connection-checker-685895c7c9-57lbv"
Nov 02 01:31:36.210103 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.210088    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"]
Nov 02 01:31:36.210216 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.210204    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.218762 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.218743    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.218844 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.218833    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.224468 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.224449    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"]
Nov 02 01:31:36.224592 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.224577    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.244478 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.244450    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"]
Nov 02 01:31:36.244767 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.244727    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.260592 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.260573    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"]
Nov 02 01:31:36.260697 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.260679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.274828 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.274808    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57b8676cdd-mw579"]
Nov 02 01:31:36.275087 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.275042    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:36.281385 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.280230    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.281385 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.280330    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.281385 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.280907    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.281385 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.281295    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.281643 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.281428    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 01:31:36.283252 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.282790    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.283252 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.282797    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 01:31:36.285275 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.285252    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 01:31:36.286030 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.286012    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.299027 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.299005    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-799447c5f7-7ffhs"]
Nov 02 01:31:36.299159 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.299145    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.306516 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kn5\" (UniqueName: \"kubernetes.io/projected/230459c8-baa4-47cc-9cc2-73f0535fb650-kube-api-access-r8kn5\") pod \"kas-connection-checker-685895c7c9-xkcjd\" (UID: \"230459c8-baa4-47cc-9cc2-73f0535fb650\") " pod="kube-system/kas-connection-checker-685895c7c9-xkcjd"
Nov 02 01:31:36.306608 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306526    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mxgq\" (UniqueName: \"kubernetes.io/projected/f3c0e9d9-eda0-4e8f-8677-0bd4406243a4-kube-api-access-4mxgq\") pod \"volume-data-source-validator-58c9f5f46d-888h6\" (UID: \"f3c0e9d9-eda0-4e8f-8677-0bd4406243a4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"
Nov 02 01:31:36.306608 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306546    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.306608 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt47v\" (UniqueName: \"kubernetes.io/projected/f74b04ce-df4a-46d3-9eb5-52464d2b5e50-kube-api-access-jt47v\") pod \"kas-connection-checker-685895c7c9-57lbv\" (UID: \"f74b04ce-df4a-46d3-9eb5-52464d2b5e50\") " pod="kube-system/kas-connection-checker-685895c7c9-57lbv"
Nov 02 01:31:36.306608 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-service-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.306757 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-tmp\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.306819 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-snapshots\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.306868 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306828    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.306868 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-config\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.306957 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306873    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1284f23-412e-4a65-970e-25ec56f3da4a-serving-cert\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.306957 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqp2\" (UniqueName: \"kubernetes.io/projected/5826852c-0e9b-4031-8213-94051d31c42c-kube-api-access-fjqp2\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.307040 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.306986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5dl\" (UniqueName: \"kubernetes.io/projected/5c414b43-ca48-4f3e-a38a-c82f2cc27283-kube-api-access-wq5dl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.307109 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307039    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-default-certificate\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.307109 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307082    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c414b43-ca48-4f3e-a38a-c82f2cc27283-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.307248 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307127    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26929d97-925d-4839-9bd7-e0fc4b3373a0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:36.307248 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307145    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:36.307248 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307161    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.307248 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b9vc\" (UniqueName: \"kubernetes.io/projected/c1284f23-412e-4a65-970e-25ec56f3da4a-kube-api-access-6b9vc\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.307248 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307235    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42z6t\" (UniqueName: \"kubernetes.io/projected/277a3a65-7e29-4cb7-a913-d364040935b0-kube-api-access-42z6t\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.307463 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307268    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c414b43-ca48-4f3e-a38a-c82f2cc27283-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.307463 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7xqs\" (UniqueName: \"kubernetes.io/projected/444113a1-c287-4a62-98ba-7b88fe123732-kube-api-access-x7xqs\") pod \"kas-connection-checker-685895c7c9-5tvxk\" (UID: \"444113a1-c287-4a62-98ba-7b88fe123732\") " pod="kube-system/kas-connection-checker-685895c7c9-5tvxk"
Nov 02 01:31:36.307463 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/277a3a65-7e29-4cb7-a913-d364040935b0-serving-cert\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.307463 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307364    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.307463 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.307619 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307468    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.307619 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-stats-auth\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.307619 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.307550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgxv\" (UniqueName: \"kubernetes.io/projected/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-kube-api-access-7tgxv\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.321274 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.321252    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"]
Nov 02 01:31:36.321385 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.321370    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.330936 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.330919    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 01:31:36.331259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.331236    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 01:31:36.331361 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.331261    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 01:31:36.331361 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.331273    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 01:31:36.331361 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.331271    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 01:31:36.331361 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.331297    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 01:31:36.331361 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.331305    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 01:31:36.331361 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.331271    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 01:31:36.331361 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.331332    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 01:31:36.331584 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.331375    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 01:31:36.334045 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.334027    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5tjm5"]
Nov 02 01:31:36.334145 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.334133    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"
Nov 02 01:31:36.357259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.357231    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sj5db"]
Nov 02 01:31:36.357368 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.357346    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:36.369589 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.369565    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kn5\" (UniqueName: \"kubernetes.io/projected/230459c8-baa4-47cc-9cc2-73f0535fb650-kube-api-access-r8kn5\") pod \"kas-connection-checker-685895c7c9-xkcjd\" (UID: \"230459c8-baa4-47cc-9cc2-73f0535fb650\") " pod="kube-system/kas-connection-checker-685895c7c9-xkcjd"
Nov 02 01:31:36.369701 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.369611    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7xqs\" (UniqueName: \"kubernetes.io/projected/444113a1-c287-4a62-98ba-7b88fe123732-kube-api-access-x7xqs\") pod \"kas-connection-checker-685895c7c9-5tvxk\" (UID: \"444113a1-c287-4a62-98ba-7b88fe123732\") " pod="kube-system/kas-connection-checker-685895c7c9-5tvxk"
Nov 02 01:31:36.369701 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.369661    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt47v\" (UniqueName: \"kubernetes.io/projected/f74b04ce-df4a-46d3-9eb5-52464d2b5e50-kube-api-access-jt47v\") pod \"kas-connection-checker-685895c7c9-57lbv\" (UID: \"f74b04ce-df4a-46d3-9eb5-52464d2b5e50\") " pod="kube-system/kas-connection-checker-685895c7c9-57lbv"
Nov 02 01:31:36.377314 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377282    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.377314 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377290    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.377454 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377333    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 01:31:36.377454 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377340    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 01:31:36.377454 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377361    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 01:31:36.377454 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377342    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.377454 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377338    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.377643 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377510    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 01:31:36.377643 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377524    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.377643 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377543    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.377643 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377560    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.377643 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.377592    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.378094 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378078    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5tvxk"]
Nov 02 01:31:36.378166 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378117    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xkcjd"]
Nov 02 01:31:36.378166 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378143    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-57lbv"]
Nov 02 01:31:36.378259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378166    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"]
Nov 02 01:31:36.378259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378167    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.378259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378187    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"]
Nov 02 01:31:36.378259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378208    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"]
Nov 02 01:31:36.378259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378230    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bv7fm"]
Nov 02 01:31:36.378259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378251    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"]
Nov 02 01:31:36.378540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378276    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"]
Nov 02 01:31:36.378540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378299    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sj5db"]
Nov 02 01:31:36.378540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378325    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"]
Nov 02 01:31:36.378540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378346    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799447c5f7-7ffhs"]
Nov 02 01:31:36.378540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378370    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"]
Nov 02 01:31:36.378540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378392    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"]
Nov 02 01:31:36.378540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378417    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5tjm5"]
Nov 02 01:31:36.378540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378440    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b8676cdd-mw579"]
Nov 02 01:31:36.378540 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378463    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7948cc5b95-b87qg"]
Nov 02 01:31:36.378852 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378758    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 01:31:36.378950 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.378935    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 01:31:36.386850 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.386835    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.386850 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.386841    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.403962 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.403896    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 01:31:36.403962 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.403903    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 01:31:36.403962 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.403921    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lljjm\""
Nov 02 01:31:36.403962 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.403931    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 01:31:36.408203 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-stats-auth\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.408286 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408221    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgxv\" (UniqueName: \"kubernetes.io/projected/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-kube-api-access-7tgxv\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.408286 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408241    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjcgz\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.408286 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408261    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.408418 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408290    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnvdl\" (UniqueName: \"kubernetes.io/projected/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-kube-api-access-cnvdl\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.408418 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408331    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ppz\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.408418 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-tmp\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.408418 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-snapshots\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.408592 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.408592 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-default-certificate\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.408592 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408597    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.408609    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-config\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.408664    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:36.908644141 +0000 UTC m=+31.674273411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408763    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-tmp\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408781    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.408899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.409085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409082    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-snapshots\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409050    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-serving-cert\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409260    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b9vc\" (UniqueName: \"kubernetes.io/projected/c1284f23-412e-4a65-970e-25ec56f3da4a-kube-api-access-6b9vc\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlm9\" (UniqueName: \"kubernetes.io/projected/944b4b8f-a639-4153-a4f7-3dc0e5691225-kube-api-access-5hlm9\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409391    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/277a3a65-7e29-4cb7-a913-d364040935b0-serving-cert\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409398    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-config\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409479    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.409547 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409583    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.409618    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:36.909600967 +0000 UTC m=+31.675230238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409639    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5dl\" (UniqueName: \"kubernetes.io/projected/5c414b43-ca48-4f3e-a38a-c82f2cc27283-kube-api-access-wq5dl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.409660    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409678    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.409711    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:36.909695063 +0000 UTC m=+31.675324311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxgq\" (UniqueName: \"kubernetes.io/projected/f3c0e9d9-eda0-4e8f-8677-0bd4406243a4-kube-api-access-4mxgq\") pod \"volume-data-source-validator-58c9f5f46d-888h6\" (UID: \"f3c0e9d9-eda0-4e8f-8677-0bd4406243a4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-service-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1284f23-412e-4a65-970e-25ec56f3da4a-serving-cert\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409825    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqp2\" (UniqueName: \"kubernetes.io/projected/5826852c-0e9b-4031-8213-94051d31c42c-kube-api-access-fjqp2\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c414b43-ca48-4f3e-a38a-c82f2cc27283-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409886    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26929d97-925d-4839-9bd7-e0fc4b3373a0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:36.410115 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.409955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410000    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410037    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42z6t\" (UniqueName: \"kubernetes.io/projected/277a3a65-7e29-4cb7-a913-d364040935b0-kube-api-access-42z6t\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410144    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c414b43-ca48-4f3e-a38a-c82f2cc27283-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410234    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410260    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncsgc\" (UniqueName: \"kubernetes.io/projected/1b3aaf5a-58ac-4fa6-846e-0764a4803577-kube-api-access-ncsgc\") pod \"network-check-source-6f78d59f99-mvlvw\" (UID: \"1b3aaf5a-58ac-4fa6-846e-0764a4803577\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410310    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-config\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.410413    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.410458    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:36.910444428 +0000 UTC m=+31.676073700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410742    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-service-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410749    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.410774 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410772    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26929d97-925d-4839-9bd7-e0fc4b3373a0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:36.411356 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.410903    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 01:31:36.411356 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.411039    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xjcn9\""
Nov 02 01:31:36.411661 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.411598    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c414b43-ca48-4f3e-a38a-c82f2cc27283-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.412009 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.411991    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-stats-auth\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.412146 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.412102    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-default-certificate\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.412961 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.412944    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/277a3a65-7e29-4cb7-a913-d364040935b0-serving-cert\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.413048 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.413032    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c414b43-ca48-4f3e-a38a-c82f2cc27283-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.413136 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.413097    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1284f23-412e-4a65-970e-25ec56f3da4a-serving-cert\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.431114 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.431088    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 01:31:36.431217 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.431177    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6csd8\""
Nov 02 01:31:36.436296 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.436273    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-57lbv"
Nov 02 01:31:36.446421 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.446400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5tvxk"
Nov 02 01:31:36.460595 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.460569    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xkcjd"
Nov 02 01:31:36.471123 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.471096    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgxv\" (UniqueName: \"kubernetes.io/projected/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-kube-api-access-7tgxv\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.471277 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.471257    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b9vc\" (UniqueName: \"kubernetes.io/projected/c1284f23-412e-4a65-970e-25ec56f3da4a-kube-api-access-6b9vc\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.471985 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.471963    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42z6t\" (UniqueName: \"kubernetes.io/projected/277a3a65-7e29-4cb7-a913-d364040935b0-kube-api-access-42z6t\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.472083 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.471970    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxgq\" (UniqueName: \"kubernetes.io/projected/f3c0e9d9-eda0-4e8f-8677-0bd4406243a4-kube-api-access-4mxgq\") pod \"volume-data-source-validator-58c9f5f46d-888h6\" (UID: \"f3c0e9d9-eda0-4e8f-8677-0bd4406243a4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"
Nov 02 01:31:36.472143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.472134    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5dl\" (UniqueName: \"kubernetes.io/projected/5c414b43-ca48-4f3e-a38a-c82f2cc27283-kube-api-access-wq5dl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.472436 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.472418    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqp2\" (UniqueName: \"kubernetes.io/projected/5826852c-0e9b-4031-8213-94051d31c42c-kube-api-access-fjqp2\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.477897 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.477846    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 01:31:36.477897 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.477864    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 01:31:36.477897 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.477875    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 01:31:36.486133 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.486108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"
Nov 02 01:31:36.500214 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.500185    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:36.511603 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:36.511603 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.511788 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511628    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-config-volume\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.511788 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511658    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.511788 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.511788 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511705    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p22\" (UniqueName: \"kubernetes.io/projected/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-kube-api-access-v4p22\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.511788 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511740    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.511788 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.511958 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-serving-cert\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.511958 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlm9\" (UniqueName: \"kubernetes.io/projected/944b4b8f-a639-4153-a4f7-3dc0e5691225-kube-api-access-5hlm9\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:36.511958 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511878    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.511958 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511906    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.512086 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511962    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77d4f2e6-e448-4f1e-b817-50cb130f99c1-cert\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:36.512086 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.511991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.512086 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfl22\" (UniqueName: \"kubernetes.io/projected/77d4f2e6-e448-4f1e-b817-50cb130f99c1-kube-api-access-gfl22\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:36.512187 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.512187 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512128    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.512187 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512156    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.512187 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsgc\" (UniqueName: \"kubernetes.io/projected/1b3aaf5a-58ac-4fa6-846e-0764a4803577-kube-api-access-ncsgc\") pod \"network-check-source-6f78d59f99-mvlvw\" (UID: \"1b3aaf5a-58ac-4fa6-846e-0764a4803577\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"
Nov 02 01:31:36.512316 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.512316 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512245    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.512316 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-config\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.512316 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512301    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-tmp-dir\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.512438 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjcgz\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.512438 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnvdl\" (UniqueName: \"kubernetes.io/projected/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-kube-api-access-cnvdl\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.512438 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512426    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ppz\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.512528 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.512528 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512489    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:36.512606 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.512590    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:36.512643 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:36.512643 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512627    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.512705 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.512682    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:36.512705 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.512692    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:36.512768 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.512743    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:37.012726211 +0000 UTC m=+31.778355459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:36.512948 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512926    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.513035 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.512985    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.513529 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.513452    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-config\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.513529 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.513546    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:37.013526183 +0000 UTC m=+31.779155433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:36.513804 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.513707    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:36.513804 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.513720    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:36.513804 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.513762    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:37.013747132 +0000 UTC m=+31.779376379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:36.514015 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.513991    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.516103 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.514955    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.516103 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.515380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.516387 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.516328    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.516469 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.516445    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.516795 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.516771    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.516976 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.516957    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.518323 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.518291    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-serving-cert\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.519692 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.519669    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.556602 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.556118    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:36.567194 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.567088    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:36.590135 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.590034    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsgc\" (UniqueName: \"kubernetes.io/projected/1b3aaf5a-58ac-4fa6-846e-0764a4803577-kube-api-access-ncsgc\") pod \"network-check-source-6f78d59f99-mvlvw\" (UID: \"1b3aaf5a-58ac-4fa6-846e-0764a4803577\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"
Nov 02 01:31:36.590749 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.590691    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnvdl\" (UniqueName: \"kubernetes.io/projected/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-kube-api-access-cnvdl\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.591454 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.591403    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.592374 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.592334    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjcgz\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:36.592973 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.592931    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.593635 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.593592    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlm9\" (UniqueName: \"kubernetes.io/projected/944b4b8f-a639-4153-a4f7-3dc0e5691225-kube-api-access-5hlm9\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:36.596548 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.596399    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ppz\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.616798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-tmp-dir\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.616879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.616927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-config-volume\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.616954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p22\" (UniqueName: \"kubernetes.io/projected/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-kube-api-access-v4p22\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.617080    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.617158    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:37.117142518 +0000 UTC m=+31.882771763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.617355    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-tmp-dir\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.617438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77d4f2e6-e448-4f1e-b817-50cb130f99c1-cert\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.617463    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfl22\" (UniqueName: \"kubernetes.io/projected/77d4f2e6-e448-4f1e-b817-50cb130f99c1-kube-api-access-gfl22\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:36.618031 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.617606    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-config-volume\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.622369 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.622326    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77d4f2e6-e448-4f1e-b817-50cb130f99c1-cert\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:36.635333 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.635300    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:36.635333 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.635335    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:36.679564 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.679537    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p22\" (UniqueName: \"kubernetes.io/projected/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-kube-api-access-v4p22\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:36.680256 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.679902    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfl22\" (UniqueName: \"kubernetes.io/projected/77d4f2e6-e448-4f1e-b817-50cb130f99c1-kube-api-access-gfl22\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:36.688816 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.688791    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:31:36.688949 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.688929    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-42sf2\""
Nov 02 01:31:36.689094 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.689044    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m64v5\""
Nov 02 01:31:36.691081 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.691047    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:36.708866 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.708845    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"
Nov 02 01:31:36.877816 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.877789    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-57lbv"]
Nov 02 01:31:36.880946 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:36.880903    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74b04ce_df4a_46d3_9eb5_52464d2b5e50.slice/crio-90e97ab6e2ab415aba3c2edd70f326729b73a49ac6767006a660bb39289edea3 WatchSource:0}: Error finding container 90e97ab6e2ab415aba3c2edd70f326729b73a49ac6767006a660bb39289edea3: Status 404 returned error can't find the container with id 90e97ab6e2ab415aba3c2edd70f326729b73a49ac6767006a660bb39289edea3
Nov 02 01:31:36.882376 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.882357    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5tvxk"]
Nov 02 01:31:36.884684 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:36.884667    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444113a1_c287_4a62_98ba_7b88fe123732.slice/crio-5d0c5618fa8f8cdb5d95e8c66bd16de3a9b5ce06e3ec5403f920bc086ab55cc3 WatchSource:0}: Error finding container 5d0c5618fa8f8cdb5d95e8c66bd16de3a9b5ce06e3ec5403f920bc086ab55cc3: Status 404 returned error can't find the container with id 5d0c5618fa8f8cdb5d95e8c66bd16de3a9b5ce06e3ec5403f920bc086ab55cc3
Nov 02 01:31:36.889316 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.889292    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:36.892307 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.892284    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xkcjd"]
Nov 02 01:31:36.896682 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:36.895528    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230459c8_baa4_47cc_9cc2_73f0535fb650.slice/crio-3c7fd19fba799063e1e9060bd7b5717b7291455494a28b9c617952a55e170cab WatchSource:0}: Error finding container 3c7fd19fba799063e1e9060bd7b5717b7291455494a28b9c617952a55e170cab: Status 404 returned error can't find the container with id 3c7fd19fba799063e1e9060bd7b5717b7291455494a28b9c617952a55e170cab
Nov 02 01:31:36.920637 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.920609    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:36.920748 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.920680    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:36.920804 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.920759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.920804 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.920776    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:36.920894 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.920848    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:37.920827679 +0000 UTC m=+32.686456941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:36.920960 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.920900    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:36.920960 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.920923    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:36.920960 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.920949    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:37.920935044 +0000 UTC m=+32.686564312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:36.921131 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.921006    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:36.921131 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.921040    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:37.921025094 +0000 UTC m=+32.686654364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:36.921131 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:36.921075    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:37.921049918 +0000 UTC m=+32.686679168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:36.973453 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.973374    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bv7fm"]
Nov 02 01:31:36.979694 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.979532    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"]
Nov 02 01:31:36.982206 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:36.982088    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"]
Nov 02 01:31:36.985672 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:36.985650    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c0e9d9_eda0_4e8f_8677_0bd4406243a4.slice/crio-60d702aa63de9a0aa6b1ca5e905eceb16f2afc3af64eecafbace041b03e876b1 WatchSource:0}: Error finding container 60d702aa63de9a0aa6b1ca5e905eceb16f2afc3af64eecafbace041b03e876b1: Status 404 returned error can't find the container with id 60d702aa63de9a0aa6b1ca5e905eceb16f2afc3af64eecafbace041b03e876b1
Nov 02 01:31:37.006924 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:37.005146    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c414b43_ca48_4f3e_a38a_c82f2cc27283.slice/crio-e0c7a4d9f829aac640f265a026904daae2465fb7ea4da328207038ef7076aa72 WatchSource:0}: Error finding container e0c7a4d9f829aac640f265a026904daae2465fb7ea4da328207038ef7076aa72: Status 404 returned error can't find the container with id e0c7a4d9f829aac640f265a026904daae2465fb7ea4da328207038ef7076aa72
Nov 02 01:31:37.015314 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.010770    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"]
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.026769    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.026872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.026943    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.027160    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.027175    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.027231    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:38.027214356 +0000 UTC m=+32.792843617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.027292    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.027301    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.027328    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:38.027319286 +0000 UTC m=+32.792948547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.027387    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:37.032482 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.027419    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:38.027408876 +0000 UTC m=+32.793038137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:37.034905 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:37.033725    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277a3a65_7e29_4cb7_a913_d364040935b0.slice/crio-805472184fa800d13999646428a1783a4d83d89725ce7c11d5392d12b3d378b1 WatchSource:0}: Error finding container 805472184fa800d13999646428a1783a4d83d89725ce7c11d5392d12b3d378b1: Status 404 returned error can't find the container with id 805472184fa800d13999646428a1783a4d83d89725ce7c11d5392d12b3d378b1
Nov 02 01:31:37.092199 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.090855    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"]
Nov 02 01:31:37.092199 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.092178    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5tjm5"]
Nov 02 01:31:37.092628 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:37.092603    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3aaf5a_58ac_4fa6_846e_0764a4803577.slice/crio-02b5e9f0603c323711292c80ad84433b43389c690fcdb3a6263a61a3a9cc9c3c WatchSource:0}: Error finding container 02b5e9f0603c323711292c80ad84433b43389c690fcdb3a6263a61a3a9cc9c3c: Status 404 returned error can't find the container with id 02b5e9f0603c323711292c80ad84433b43389c690fcdb3a6263a61a3a9cc9c3c
Nov 02 01:31:37.098027 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:37.098007    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d4f2e6_e448_4f1e_b817_50cb130f99c1.slice/crio-e368bf4a9b539e4b1a3f41970c9677b3ec8fb8d678a615afbbb5a435463eb541 WatchSource:0}: Error finding container e368bf4a9b539e4b1a3f41970c9677b3ec8fb8d678a615afbbb5a435463eb541: Status 404 returned error can't find the container with id e368bf4a9b539e4b1a3f41970c9677b3ec8fb8d678a615afbbb5a435463eb541
Nov 02 01:31:37.128177 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.128148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:37.128349 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.128301    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:37.128411 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.128360    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:38.128346487 +0000 UTC m=+32.893975732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:37.321406 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.317664    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"]
Nov 02 01:31:37.635389 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.635312    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:37.688658 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.688625    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:31:37.688658 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.688628    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fxpjk\""
Nov 02 01:31:37.869645 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.869614    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5tjm5" event={"ID":"77d4f2e6-e448-4f1e-b817-50cb130f99c1","Type":"ContainerStarted","Data":"e368bf4a9b539e4b1a3f41970c9677b3ec8fb8d678a615afbbb5a435463eb541"}
Nov 02 01:31:37.870758 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.870735    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv" event={"ID":"5c414b43-ca48-4f3e-a38a-c82f2cc27283","Type":"ContainerStarted","Data":"e0c7a4d9f829aac640f265a026904daae2465fb7ea4da328207038ef7076aa72"}
Nov 02 01:31:37.871757 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.871738    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bv7fm" event={"ID":"c1284f23-412e-4a65-970e-25ec56f3da4a","Type":"ContainerStarted","Data":"e26a637c97de91d54768fa1a41f450110d39e6f07b6ba622e7245b6e9bf69178"}
Nov 02 01:31:37.872835 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.872816    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6" event={"ID":"f3c0e9d9-eda0-4e8f-8677-0bd4406243a4","Type":"ContainerStarted","Data":"60d702aa63de9a0aa6b1ca5e905eceb16f2afc3af64eecafbace041b03e876b1"}
Nov 02 01:31:37.873854 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.873833    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw" event={"ID":"1b3aaf5a-58ac-4fa6-846e-0764a4803577","Type":"ContainerStarted","Data":"02b5e9f0603c323711292c80ad84433b43389c690fcdb3a6263a61a3a9cc9c3c"}
Nov 02 01:31:37.875189 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.875168    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xkcjd" event={"ID":"230459c8-baa4-47cc-9cc2-73f0535fb650","Type":"ContainerStarted","Data":"14f1ddf73439454bc98d346d3e78e890bd0b3f35cc9eaa240718676353d59a5b"}
Nov 02 01:31:37.875272 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.875204    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xkcjd" event={"ID":"230459c8-baa4-47cc-9cc2-73f0535fb650","Type":"ContainerStarted","Data":"3c7fd19fba799063e1e9060bd7b5717b7291455494a28b9c617952a55e170cab"}
Nov 02 01:31:37.876558 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.876533    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-57lbv" event={"ID":"f74b04ce-df4a-46d3-9eb5-52464d2b5e50","Type":"ContainerStarted","Data":"ec45d60b7a1e6a95b64572400cbd16907b51bac0066024939e02b2ee857e3bf6"}
Nov 02 01:31:37.876655 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.876572    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-57lbv" event={"ID":"f74b04ce-df4a-46d3-9eb5-52464d2b5e50","Type":"ContainerStarted","Data":"90e97ab6e2ab415aba3c2edd70f326729b73a49ac6767006a660bb39289edea3"}
Nov 02 01:31:37.877595 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.877579    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw" event={"ID":"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4","Type":"ContainerStarted","Data":"d2a89733b88aea8224676f032630116b71e58a415208c3ff38c23b1668f2bdab"}
Nov 02 01:31:37.878674 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.878656    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" event={"ID":"277a3a65-7e29-4cb7-a913-d364040935b0","Type":"ContainerStarted","Data":"805472184fa800d13999646428a1783a4d83d89725ce7c11d5392d12b3d378b1"}
Nov 02 01:31:37.880028 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.880009    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5tvxk" event={"ID":"444113a1-c287-4a62-98ba-7b88fe123732","Type":"ContainerStarted","Data":"512de28843eb5cb05cfd4899cadf289a40e68729c67ae4a0ef1b564ed6414446"}
Nov 02 01:31:37.880103 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.880042    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5tvxk" event={"ID":"444113a1-c287-4a62-98ba-7b88fe123732","Type":"ContainerStarted","Data":"5d0c5618fa8f8cdb5d95e8c66bd16de3a9b5ce06e3ec5403f920bc086ab55cc3"}
Nov 02 01:31:37.935441 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.935355    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:37.935441 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.935389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:37.935441 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.935423    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:37.935689 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.935524    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:37.935689 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.935525    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:37.935689 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.935556    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:39.935535783 +0000 UTC m=+34.701165033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:37.935689 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.935586    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:39.935569887 +0000 UTC m=+34.701199134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:37.935689 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:37.935639    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:37.935689 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.935652    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:39.935640217 +0000 UTC m=+34.701269476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:37.935689 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.935681    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:37.935911 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:37.935711    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:39.935703526 +0000 UTC m=+34.701332772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:38.036933 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.036900    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:38.037133 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.036963    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:38.037133 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.037099    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:38.039088 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.038488    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:38.039088 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.038556    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:40.03853662 +0000 UTC m=+34.804165871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:38.039088 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.038622    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:38.039088 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.038633    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:38.039088 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.038666    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:40.038654187 +0000 UTC m=+34.804283433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:38.039344 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.039183    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:38.039344 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.039198    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:38.039344 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.039257    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:40.039243062 +0000 UTC m=+34.804872319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:38.138572 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.138539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:38.138760 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.138727    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:38.138822 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.138804    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:40.138784413 +0000 UTC m=+34.904413662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:38.263086 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.262376    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xkcjd" podStartSLOduration=411.262356699 podStartE2EDuration="6m51.262356699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:38.261817132 +0000 UTC m=+33.027446401" watchObservedRunningTime="2026-11-02 01:31:38.262356699 +0000 UTC m=+33.027985987"
Nov 02 01:31:38.501095 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.499528    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5tvxk" podStartSLOduration=411.499506395 podStartE2EDuration="6m51.499506395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:38.498916703 +0000 UTC m=+33.264545968" watchObservedRunningTime="2026-11-02 01:31:38.499506395 +0000 UTC m=+33.265135664"
Nov 02 01:31:38.501095 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.499696    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-57lbv" podStartSLOduration=411.499689611 podStartE2EDuration="6m51.499689611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:38.381806463 +0000 UTC m=+33.147435733" watchObservedRunningTime="2026-11-02 01:31:38.499689611 +0000 UTC m=+33.265318879"
Nov 02 01:31:38.647552 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.647466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:38.647717 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.647622    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:38.647811 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.647795    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:31:38.647876 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:38.647865    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:32:10.647845695 +0000 UTC m=+65.413474944 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : secret "metrics-daemon-secret" not found
Nov 02 01:31:38.665969 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.659004    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:38.749166 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.748403    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:38.752938 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.752861    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:38.760408 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.760195    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:38.779478 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.779248    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:38.921345 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:38.920503    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:39.147799 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.147767    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:39.212442 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.212299    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nfs6f"]
Nov 02 01:31:39.223581 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.222875    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hqwck"]
Nov 02 01:31:39.230001 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:39.229906    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bd96c5_7bd3_4b79_847c_fd7f7866e231.slice/crio-0ee4f82ebcf21ee10a638f51a8b3db2081ed076c0999ad91434839929af869c8 WatchSource:0}: Error finding container 0ee4f82ebcf21ee10a638f51a8b3db2081ed076c0999ad91434839929af869c8: Status 404 returned error can't find the container with id 0ee4f82ebcf21ee10a638f51a8b3db2081ed076c0999ad91434839929af869c8
Nov 02 01:31:39.325661 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.325601    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:39.896760 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.896716    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nfs6f" event={"ID":"447b6716-29a3-4000-8df4-022f92de0e23","Type":"ContainerStarted","Data":"4a8dc9f2139340a5111148a200f3e2b4d24896cf51c6fe2b265e612683e14e42"}
Nov 02 01:31:39.899905 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.899865    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hqwck" event={"ID":"08bd96c5-7bd3-4b79-847c-fd7f7866e231","Type":"ContainerStarted","Data":"0ee4f82ebcf21ee10a638f51a8b3db2081ed076c0999ad91434839929af869c8"}
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.963640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.963727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.963841    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:39.963871    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:39.964068    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:39.964218    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:43.964199468 +0000 UTC m=+38.729828733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:39.964631    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:39.964683    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:43.964671937 +0000 UTC m=+38.730301196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:39.964725    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:39.964748    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:43.96474034 +0000 UTC m=+38.730369599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:39.964858 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:39.964792    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:43.964785989 +0000 UTC m=+38.730415248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:40.065188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:40.065336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:40.065409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.065562    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.065639    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:44.065619476 +0000 UTC m=+38.831248742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.065712    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.065722    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.065754    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:44.065744032 +0000 UTC m=+38.831373284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.065805    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.065815    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:40.065883 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.065843    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:44.065833477 +0000 UTC m=+38.831462737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:40.167010 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:40.166653    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:40.167010 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.166906    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:40.167010 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:40.166971    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:44.166952654 +0000 UTC m=+38.932581920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:40.519687 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:40.519553    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:40.700185 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:40.700151    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:41.914526 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:41.912867    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6" event={"ID":"f3c0e9d9-eda0-4e8f-8677-0bd4406243a4","Type":"ContainerStarted","Data":"a9eeee92efd49397005826dd55e63e616eb62b266192719a6286709a8963433e"}
Nov 02 01:31:41.915522 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:41.915489    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" event={"ID":"277a3a65-7e29-4cb7-a913-d364040935b0","Type":"ContainerStarted","Data":"a26a1c0a69176754ec3e0e4105a548951cdfc19d6407c96ef28ea04f1b0d3a9c"}
Nov 02 01:31:41.917378 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:41.917355    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:41.917603 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:41.917559    2573 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-2m8ck container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body=
Nov 02 01:31:41.917676 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:41.917606    2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused"
Nov 02 01:31:42.032000 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:42.031889    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6" podStartSLOduration=385.242349309 podStartE2EDuration="6m29.031871949s" totalImagesPullingTime="3.78952264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:42.030302497 +0000 UTC m=+36.795931768" watchObservedRunningTime="2026-11-02 01:31:42.031871949 +0000 UTC m=+36.797501218"
Nov 02 01:31:42.149436 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:42.149370    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podStartSLOduration=384.445619569 podStartE2EDuration="6m29.149349961s" totalImagesPullingTime="4.703730392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:42.148247003 +0000 UTC m=+36.913876271" watchObservedRunningTime="2026-11-02 01:31:42.149349961 +0000 UTC m=+36.914979230"
Nov 02 01:31:42.220682 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:42.220606    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:42.396278 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:42.396190    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:42.923158 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:42.922339    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bv7fm" event={"ID":"c1284f23-412e-4a65-970e-25ec56f3da4a","Type":"ContainerStarted","Data":"98ba3ff976c768ca86ad5fa72e0f149cb94550654e9bc4c3c1f5743a62803514"}
Nov 02 01:31:42.929180 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:42.929132    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/0.log"
Nov 02 01:31:42.929322 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:42.929180    2573 generic.go:356] "Generic (PLEG): container finished" podID="277a3a65-7e29-4cb7-a913-d364040935b0" containerID="a26a1c0a69176754ec3e0e4105a548951cdfc19d6407c96ef28ea04f1b0d3a9c" exitCode=255
Nov 02 01:31:42.929322 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:42.929256    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" event={"ID":"277a3a65-7e29-4cb7-a913-d364040935b0","Type":"ContainerDied","Data":"a26a1c0a69176754ec3e0e4105a548951cdfc19d6407c96ef28ea04f1b0d3a9c"}
Nov 02 01:31:42.929660 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:42.929641    2573 scope.go:122] "RemoveContainer" containerID="a26a1c0a69176754ec3e0e4105a548951cdfc19d6407c96ef28ea04f1b0d3a9c"
Nov 02 01:31:43.043814 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.043743    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-bv7fm" podStartSLOduration=384.557549561 podStartE2EDuration="6m30.043723417s" totalImagesPullingTime="5.486173856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:43.042764343 +0000 UTC m=+37.808393610" watchObservedRunningTime="2026-11-02 01:31:43.043723417 +0000 UTC m=+37.809352687"
Nov 02 01:31:43.938703 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.938659    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5tjm5" event={"ID":"77d4f2e6-e448-4f1e-b817-50cb130f99c1","Type":"ContainerStarted","Data":"2de201b7fc768737ef6399367e967370303e69dc15b3f29b7dd930805f3106d3"}
Nov 02 01:31:43.943333 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.942830    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv" event={"ID":"5c414b43-ca48-4f3e-a38a-c82f2cc27283","Type":"ContainerStarted","Data":"ebc87cfb861305311296a3adfcf5d35611ab8d871d179b738972eb12f63dc6b9"}
Nov 02 01:31:43.946001 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.945973    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw" event={"ID":"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4","Type":"ContainerStarted","Data":"aac04af2ee54f1a2da1f739fdcf76f918f47ef4eab69dfea106d6f64ef59e02d"}
Nov 02 01:31:43.949296 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.949275    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/1.log"
Nov 02 01:31:43.949868 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.949851    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/0.log"
Nov 02 01:31:43.949938 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.949884    2573 generic.go:356] "Generic (PLEG): container finished" podID="277a3a65-7e29-4cb7-a913-d364040935b0" containerID="0b2f329e986fdd8bdd7308eed4623b842d3585f482c0e1cf35ea45cc222a453f" exitCode=255
Nov 02 01:31:43.950557 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.950542    2573 scope.go:122] "RemoveContainer" containerID="0b2f329e986fdd8bdd7308eed4623b842d3585f482c0e1cf35ea45cc222a453f"
Nov 02 01:31:43.950723 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:43.950702    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m8ck_openshift-console-operator(277a3a65-7e29-4cb7-a913-d364040935b0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0"
Nov 02 01:31:43.950761 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.950748    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" event={"ID":"277a3a65-7e29-4cb7-a913-d364040935b0","Type":"ContainerDied","Data":"0b2f329e986fdd8bdd7308eed4623b842d3585f482c0e1cf35ea45cc222a453f"}
Nov 02 01:31:43.950811 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.950798    2573 scope.go:122] "RemoveContainer" containerID="a26a1c0a69176754ec3e0e4105a548951cdfc19d6407c96ef28ea04f1b0d3a9c"
Nov 02 01:31:43.954619 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.954592    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dtdwx"]
Nov 02 01:31:43.971999 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:43.971963    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.009337 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.009300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.009474 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.009360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:44.009474 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.009387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:44.009474 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.009439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.009638 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.009551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:44.009695 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.009657    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7121a072-9bc5-41fb-920e-bff50c7cd95f-data-volume\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.009747 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.009693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:44.009822 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.009808    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7121a072-9bc5-41fb-920e-bff50c7cd95f-crio-socket\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.009877 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.009845    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztztb\" (UniqueName: \"kubernetes.io/projected/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-api-access-ztztb\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.010275 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.010255    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:44.010385 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.010316    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.010299403 +0000 UTC m=+46.775928664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:44.011050 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.011028    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:44.011169 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.011150    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:44.011232 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.011152    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.011133532 +0000 UTC m=+46.776762784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:44.011232 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.011214    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.011198059 +0000 UTC m=+46.776827311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:44.011232 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.011230    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.01122055 +0000 UTC m=+46.776849801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:44.018671 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.018595    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtdwx"]
Nov 02 01:31:44.025448 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.025425    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 01:31:44.025538 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.025498    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 01:31:44.040242 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.040204    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qn9km\""
Nov 02 01:31:44.068261 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.068077    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5tjm5" podStartSLOduration=1.451727325 podStartE2EDuration="8.068035535s" totalImagesPullingTime="6.61630821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:44.061505214 +0000 UTC m=+38.827134486" watchObservedRunningTime="2026-11-02 01:31:44.068035535 +0000 UTC m=+38.833664804"
Nov 02 01:31:44.110957 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.110918    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:44.111150 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.110979    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7121a072-9bc5-41fb-920e-bff50c7cd95f-crio-socket\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.111150 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.111017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztztb\" (UniqueName: \"kubernetes.io/projected/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-api-access-ztztb\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.111150 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.111046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:44.111150 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111110    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:44.111150 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.111121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.111412 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111185    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.111163849 +0000 UTC m=+46.876793111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:44.111412 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.111212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.111412 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111227    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 01:31:44.111412 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111289    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:44.611272975 +0000 UTC m=+39.376902234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : configmap "kube-rbac-proxy" not found
Nov 02 01:31:44.111412 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.111323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:44.111412 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111336    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:44.111412 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.111362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7121a072-9bc5-41fb-920e-bff50c7cd95f-data-volume\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.111412 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111378    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:44.611363458 +0000 UTC m=+39.376992723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:44.111795 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.111658    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7121a072-9bc5-41fb-920e-bff50c7cd95f-data-volume\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.111795 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111752    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:44.111795 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111765    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:44.111932 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111801    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.111789044 +0000 UTC m=+46.877418295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:44.111932 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.111816    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7121a072-9bc5-41fb-920e-bff50c7cd95f-crio-socket\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.111932 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111864    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:44.111932 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111873    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:44.111932 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.111912    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.111901433 +0000 UTC m=+46.877530682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:44.173521 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.173478    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztztb\" (UniqueName: \"kubernetes.io/projected/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-api-access-ztztb\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.212018 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.211987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:44.212316 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.212222    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:44.212316 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.212281    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.212269551 +0000 UTC m=+46.977898797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:44.212500 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.212316    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:44.390338 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.390226    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:44.537232 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.537173    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv" podStartSLOduration=385.444724844 podStartE2EDuration="6m31.53715472s" totalImagesPullingTime="6.092429876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:44.417411479 +0000 UTC m=+39.183040746" watchObservedRunningTime="2026-11-02 01:31:44.53715472 +0000 UTC m=+39.302783989"
Nov 02 01:31:44.618452 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.618416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.618654 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.618478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:44.618654 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.618555    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 01:31:44.618654 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.618629    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:45.618609218 +0000 UTC m=+40.384238554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : configmap "kube-rbac-proxy" not found
Nov 02 01:31:44.618827 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.618657    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:44.618827 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.618727    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:45.618708118 +0000 UTC m=+40.384337369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:44.644346 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.644206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:44.824612 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.824559    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:44.827119 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.827069    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw" podStartSLOduration=385.457438214 podStartE2EDuration="6m31.827036498s" totalImagesPullingTime="6.369598284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:44.536101186 +0000 UTC m=+39.301730455" watchObservedRunningTime="2026-11-02 01:31:44.827036498 +0000 UTC m=+39.592665768"
Nov 02 01:31:44.828008 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.827985    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"]
Nov 02 01:31:44.859896 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.859856    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"
Nov 02 01:31:44.893600 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.893559    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"]
Nov 02 01:31:44.921764 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.921668    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgxf\" (UniqueName: \"kubernetes.io/projected/6c3d1051-a370-4a67-bebf-78d6cd902101-kube-api-access-lqgxf\") pod \"migrator-66df5d496b-6brtq\" (UID: \"6c3d1051-a370-4a67-bebf-78d6cd902101\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"
Nov 02 01:31:44.956275 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.956244    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/1.log"
Nov 02 01:31:44.957308 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.957043    2573 scope.go:122] "RemoveContainer" containerID="0b2f329e986fdd8bdd7308eed4623b842d3585f482c0e1cf35ea45cc222a453f"
Nov 02 01:31:44.957704 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:44.957592    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m8ck_openshift-console-operator(277a3a65-7e29-4cb7-a913-d364040935b0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0"
Nov 02 01:31:44.960279 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:44.960122    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw" event={"ID":"1b3aaf5a-58ac-4fa6-846e-0764a4803577","Type":"ContainerStarted","Data":"41024cac3b26d9d4c94531e80b897b03615dec9e1a18cfd765edca278cd3c5c9"}
Nov 02 01:31:45.001877 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.001650    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:45.001877 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.001762    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 01:31:45.001877 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.001839    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-846kb\""
Nov 02 01:31:45.004404 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.004374    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:45.023074 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.023027    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgxf\" (UniqueName: \"kubernetes.io/projected/6c3d1051-a370-4a67-bebf-78d6cd902101-kube-api-access-lqgxf\") pod \"migrator-66df5d496b-6brtq\" (UID: \"6c3d1051-a370-4a67-bebf-78d6cd902101\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"
Nov 02 01:31:45.096563 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.096510    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw" podStartSLOduration=380.685301145 podStartE2EDuration="6m28.09649113s" totalImagesPullingTime="7.411189985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:45.091387507 +0000 UTC m=+39.857016775" watchObservedRunningTime="2026-11-02 01:31:45.09649113 +0000 UTC m=+39.862120399"
Nov 02 01:31:45.096882 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.096854    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgxf\" (UniqueName: \"kubernetes.io/projected/6c3d1051-a370-4a67-bebf-78d6cd902101-kube-api-access-lqgxf\") pod \"migrator-66df5d496b-6brtq\" (UID: \"6c3d1051-a370-4a67-bebf-78d6cd902101\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"
Nov 02 01:31:45.183317 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.183238    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"
Nov 02 01:31:45.188405 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.188368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:45.390071 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.390009    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:45.572426 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.572379    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:45.576822 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.576792    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"]
Nov 02 01:31:45.581191 ip-10-0-135-33 kubenswrapper[2573]: W1102 01:31:45.581162    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3d1051_a370_4a67_bebf_78d6cd902101.slice/crio-8292077652b355c56bcdce4f29f614a97ba3380fcf87d41b3a83e1518691a626 WatchSource:0}: Error finding container 8292077652b355c56bcdce4f29f614a97ba3380fcf87d41b3a83e1518691a626: Status 404 returned error can't find the container with id 8292077652b355c56bcdce4f29f614a97ba3380fcf87d41b3a83e1518691a626
Nov 02 01:31:45.633997 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.633964    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:45.634198 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.634041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:45.634198 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:45.634115    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 01:31:45.634198 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:45.634198    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:47.634179281 +0000 UTC m=+42.399808564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : configmap "kube-rbac-proxy" not found
Nov 02 01:31:45.634364 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:45.634222    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:45.634364 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:45.634294    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:47.634274432 +0000 UTC m=+42.399903693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:45.746010 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.745926    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:45.964143 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.964102    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq" event={"ID":"6c3d1051-a370-4a67-bebf-78d6cd902101","Type":"ContainerStarted","Data":"8292077652b355c56bcdce4f29f614a97ba3380fcf87d41b3a83e1518691a626"}
Nov 02 01:31:45.965956 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:45.965920    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hqwck" event={"ID":"08bd96c5-7bd3-4b79-847c-fd7f7866e231","Type":"ContainerStarted","Data":"7de149608a41c8d965eda25e0f4c2346ca8adf6662229e6e8aa3cc9d29f1e7bd"}
Nov 02 01:31:46.084045 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:46.083956    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hqwck" podStartSLOduration=33.863888568 podStartE2EDuration="40.083942314s" totalImagesPullingTime="6.220053746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:46.082998599 +0000 UTC m=+40.848627868" watchObservedRunningTime="2026-11-02 01:31:46.083942314 +0000 UTC m=+40.849571582"
Nov 02 01:31:46.568085 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:46.567976    2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:46.568444 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:46.568426    2573 scope.go:122] "RemoveContainer" containerID="0b2f329e986fdd8bdd7308eed4623b842d3585f482c0e1cf35ea45cc222a453f"
Nov 02 01:31:46.568684 ip-10-0-135-33 kubenswrapper[2573]: E1102 01:31:46.568644    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m8ck_openshift-console-operator(277a3a65-7e29-4cb7-a913-d364040935b0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0"
Nov 02 01:31:46.832649 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:46.832619    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:46.971259 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:46.971223    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nfs6f" event={"ID":"447b6716-29a3-4000-8df4-022f92de0e23","Type":"ContainerStarted","Data":"71905debcf0e6600786619c99baf4feb1daf0087ea6e1be1a28f23da84bf8865"}
Nov 02 01:31:46.971748 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:46.971311    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:47.008334 ip-10-0-135-33 kubenswrapper[2573]: I1102 01:31:47.008294    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:47.030011 ip-10-0-135-33 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 01:31:47.043516 ip-10-0-135-33 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 01:31:47.043732 ip-10-0-135-33 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 01:31:47.043948 ip-10-0-135-33 systemd[1]: kubelet.service: Consumed 3.208s CPU time.
Nov 02 01:31:47.062604 ip-10-0-135-33 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:31:47.125038 ip-10-0-135-33 kubenswrapper[4912]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:31:47.125038 ip-10-0-135-33 kubenswrapper[4912]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:31:47.125038 ip-10-0-135-33 kubenswrapper[4912]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:31:47.125038 ip-10-0-135-33 kubenswrapper[4912]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:31:47.127325 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127249    4912 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:47.127325 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127325    4912 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127336    4912 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127341    4912 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127344    4912 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127347    4912 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127349    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127353    4912 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127357    4912 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127361    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127363    4912 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127367    4912 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127369    4912 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127372    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127375    4912 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127378    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127380    4912 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127383    4912 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127386    4912 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127388    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127391    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:47.127396 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127393    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127396    4912 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127399    4912 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127402    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127404    4912 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127407    4912 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127410    4912 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127413    4912 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127416    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127418    4912 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127430    4912 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127432    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127435    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127438    4912 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127441    4912 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127445    4912 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127449    4912 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127452    4912 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127455    4912 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:47.127897 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127458    4912 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127460    4912 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127463    4912 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127465    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127468    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127470    4912 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127473    4912 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127475    4912 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127478    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127481    4912 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127483    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127486    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127488    4912 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127491    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127493    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127497    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127500    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127503    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127505    4912 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:47.128378 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127508    4912 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127511    4912 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127514    4912 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127516    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127519    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127521    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127530    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127533    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127535    4912 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127538    4912 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127540    4912 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127543    4912 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127545    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127549    4912 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127553    4912 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127557    4912 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127561    4912 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127564    4912 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127567    4912 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:47.128860 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127571    4912 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127576    4912 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127579    4912 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127582    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127585    4912 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127588    4912 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127593    4912 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127597    4912 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127600    4912 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127604    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127607    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127610    4912 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127612    4912 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127615    4912 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127618    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127621    4912 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127624    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127626    4912 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127629    4912 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127631    4912 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:47.129355 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127639    4912 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127642    4912 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127645    4912 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127647    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127650    4912 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127653    4912 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127656    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.127659    4912 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128430    4912 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128437    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128440    4912 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128442    4912 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128445    4912 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128448    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128451    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128454    4912 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128456    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128459    4912 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128463    4912 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128466    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:47.129830 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128468    4912 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128471    4912 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128474    4912 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128477    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128479    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128482    4912 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128485    4912 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128487    4912 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128490    4912 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128493    4912 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128496    4912 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128498    4912 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128501    4912 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128504    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128507    4912 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128509    4912 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128512    4912 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128514    4912 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128517    4912 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128519    4912 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:47.130384 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128522    4912 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128524    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128527    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128529    4912 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128531    4912 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128534    4912 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128537    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128540    4912 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128542    4912 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128545    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128547    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128550    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128553    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128555    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128557    4912 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128560    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128562    4912 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128565    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128567    4912 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:47.130879 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128570    4912 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128573    4912 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128576    4912 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128578    4912 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128581    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128583    4912 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128586    4912 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128589    4912 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128592    4912 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128594    4912 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128597    4912 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128599    4912 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128602    4912 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128604    4912 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128607    4912 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128609    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128612    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128614    4912 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128617    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128620    4912 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:47.131375 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128622    4912 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128625    4912 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128627    4912 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128632    4912 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128637    4912 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128641    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128644    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128646    4912 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128649    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128651    4912 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128654    4912 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128658    4912 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128662    4912 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128665    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128668    4912 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128670    4912 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128673    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128675    4912 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128678    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:47.131894 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128680    4912 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128683    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128686    4912 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128688    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128691    4912 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128693    4912 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128695    4912 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128698    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128700    4912 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128703    4912 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128706    4912 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128710    4912 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128712    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128715    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128717    4912 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.128720    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129223    4912 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129229    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129232    4912 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129235    4912 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:47.132367 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129238    4912 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129241    4912 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129243    4912 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129246    4912 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129249    4912 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129251    4912 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129254    4912 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129257    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129259    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129262    4912 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129264    4912 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129267    4912 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129269    4912 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129273    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129275    4912 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129278    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129281    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129283    4912 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129286    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129288    4912 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:47.132853 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129291    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129293    4912 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129298    4912 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129301    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129303    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129306    4912 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129308    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129311    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129315    4912 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129318    4912 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129320    4912 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129323    4912 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129326    4912 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129328    4912 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129331    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129333    4912 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129336    4912 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129340    4912 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129343    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:47.133566 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129346    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129349    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129351    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129354    4912 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129356    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129359    4912 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129362    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129365    4912 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129368    4912 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129371    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129374    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129376    4912 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129379    4912 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129381    4912 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129384    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129387    4912 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129390    4912 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129392    4912 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129395    4912 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129397    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:47.134351 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129400    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129403    4912 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129406    4912 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129408    4912 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129411    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129413    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129416    4912 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129418    4912 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129421    4912 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129423    4912 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129426    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129428    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129432    4912 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129436    4912 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129439    4912 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129442    4912 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129445    4912 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129447    4912 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129450    4912 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:47.134912 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129452    4912 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129455    4912 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129458    4912 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129460    4912 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129463    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129465    4912 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129467    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129470    4912 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129473    4912 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129475    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129478    4912 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129481    4912 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129483    4912 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129486    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129489    4912 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129492    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129495    4912 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129497    4912 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129500    4912 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:47.135432 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129502    4912 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129505    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129507    4912 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129511    4912 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.129513    4912 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129601    4912 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129608    4912 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129614    4912 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129619    4912 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129623    4912 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129626    4912 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129630    4912 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129634    4912 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129637    4912 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129640    4912 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129644    4912 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129647    4912 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129651    4912 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129654    4912 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129656    4912 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129659    4912 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129662    4912 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129665    4912 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129670    4912 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:31:47.135909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129673    4912 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129676    4912 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129679    4912 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129683    4912 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129687    4912 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129690    4912 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129693    4912 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129696    4912 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129699    4912 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129702    4912 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129705    4912 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129708    4912 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129711    4912 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129715    4912 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129718    4912 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129721    4912 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129724    4912 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129727    4912 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129730    4912 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129735    4912 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129738    4912 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129741    4912 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129744    4912 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129746    4912 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:31:47.136508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129751    4912 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129754    4912 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129757    4912 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129760    4912 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129763    4912 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129766    4912 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129769    4912 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129772    4912 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129775    4912 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129777    4912 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129780    4912 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129784    4912 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129787    4912 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129790    4912 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129793    4912 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129796    4912 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129799    4912 flags.go:64] FLAG: --help="false"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129802    4912 flags.go:64] FLAG: --hostname-override="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129805    4912 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129808    4912 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129811    4912 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129814    4912 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129817    4912 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:31:47.137142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129820    4912 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129823    4912 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129826    4912 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129828    4912 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129833    4912 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129836    4912 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129839    4912 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129842    4912 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129845    4912 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129848    4912 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129852    4912 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129855    4912 flags.go:64] FLAG: --lock-file=""
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129857    4912 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129860    4912 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129864    4912 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129869    4912 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129872    4912 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129875    4912 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129878    4912 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129881    4912 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129885    4912 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129887    4912 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129890    4912 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129895    4912 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129898    4912 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:31:47.137736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129901    4912 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129904    4912 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129907    4912 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129910    4912 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129913    4912 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129916    4912 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129919    4912 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129922    4912 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129929    4912 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129932    4912 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129934    4912 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129939    4912 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129942    4912 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129945    4912 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129948    4912 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129951    4912 flags.go:64] FLAG: --port="10250"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129954    4912 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129957    4912 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b3b39b36fb2b20e6"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129961    4912 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129963    4912 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129967    4912 flags.go:64] FLAG: --register-node="true"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129971    4912 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129975    4912 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129977    4912 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129980    4912 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129983    4912 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:31:47.138354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129987    4912 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129989    4912 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129992    4912 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129996    4912 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.129998    4912 flags.go:64] FLAG: --runonce="false"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130001    4912 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130004    4912 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130007    4912 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130010    4912 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130013    4912 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130015    4912 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130018    4912 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130021    4912 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130024    4912 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130027    4912 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130030    4912 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130032    4912 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130036    4912 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130039    4912 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130042    4912 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130047    4912 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130050    4912 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130070    4912 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130075    4912 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130078    4912 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:31:47.138997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130081    4912 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130084    4912 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130088    4912 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130091    4912 flags.go:64] FLAG: --v="2"
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130096    4912 flags.go:64] FLAG: --version="false"
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130105    4912 flags.go:64] FLAG: --vmodule=""
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130110    4912 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130114    4912 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130245    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130249    4912 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130252    4912 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130255    4912 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130258    4912 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130261    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130263    4912 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130265    4912 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130268    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130271    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130274    4912 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130276    4912 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130279    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130281    4912 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:47.139629 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130284    4912 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130286    4912 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130291    4912 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130295    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130297    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130300    4912 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130302    4912 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130305    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130307    4912 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130310    4912 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130312    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130316    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130319    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130322    4912 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130325    4912 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130327    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130330    4912 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130333    4912 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130335    4912 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:47.140272 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130338    4912 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130340    4912 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130343    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130346    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130348    4912 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130351    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130354    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130356    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130359    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130361    4912 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130363    4912 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130366    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130368    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130371    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130373    4912 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130377    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130380    4912 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130383    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130385    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:47.140747 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130388    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130391    4912 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130394    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130396    4912 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130399    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130402    4912 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130406    4912 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130409    4912 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130411    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130414    4912 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130417    4912 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130419    4912 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130422    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130424    4912 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130427    4912 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130430    4912 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130432    4912 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130435    4912 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130437    4912 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130440    4912 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:47.141226 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130442    4912 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130446    4912 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130450    4912 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130453    4912 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130455    4912 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130458    4912 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130460    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130464    4912 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130470    4912 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130473    4912 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130475    4912 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130478    4912 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130481    4912 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130483    4912 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130486    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130489    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130491    4912 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130494    4912 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130497    4912 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:47.141808 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130500    4912 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130503    4912 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130505    4912 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130508    4912 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130511    4912 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130513    4912 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130515    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130518    4912 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130520    4912 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130523    4912 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130525    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130528    4912 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130530    4912 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130533    4912 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130535    4912 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.130561    4912 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:31:47.142343 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130598    4912 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130603    4912 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130606    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130609    4912 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130611    4912 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130614    4912 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130620    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130622    4912 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130625    4912 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130627    4912 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130630    4912 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130633    4912 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130635    4912 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130638    4912 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130641    4912 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130646    4912 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130650    4912 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130654    4912 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130657    4912 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:47.142809 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130660    4912 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130663    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130666    4912 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130670    4912 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130674    4912 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130677    4912 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130680    4912 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130682    4912 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130685    4912 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130687    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130690    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130692    4912 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130695    4912 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130697    4912 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130700    4912 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130703    4912 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130706    4912 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130708    4912 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130711    4912 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130715    4912 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:47.143315 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130717    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130720    4912 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130722    4912 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130725    4912 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130727    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130730    4912 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130733    4912 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130735    4912 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130739    4912 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130742    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130744    4912 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130747    4912 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130750    4912 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130753    4912 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130755    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130758    4912 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130760    4912 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130763    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130765    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130768    4912 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:47.143901 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130770    4912 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130773    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130775    4912 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130778    4912 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130780    4912 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130783    4912 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130787    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130789    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130792    4912 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130794    4912 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130796    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130800    4912 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130803    4912 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130805    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130808    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130810    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130813    4912 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130816    4912 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130818    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:47.144540 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130821    4912 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130826    4912 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130828    4912 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130831    4912 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130833    4912 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130836    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130839    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130841    4912 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130844    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130846    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130849    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130851    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130854    4912 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130856    4912 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130859    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130861    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130864    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130866    4912 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130869    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:47.145165 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130871    4912 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130874    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130876    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130879    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130881    4912 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130886    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130889    4912 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130891    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.130895    4912 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.136207    4912 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.136224    4912 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.136248    4912 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.136253    4912 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136322    4912 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136327    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136330    4912 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136333    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136336    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136339    4912 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:47.145621 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136342    4912 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136346    4912 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136350    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136354    4912 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136356    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136359    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136362    4912 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136365    4912 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136367    4912 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136370    4912 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136373    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136376    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136378    4912 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136381    4912 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136383    4912 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136386    4912 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136389    4912 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136392    4912 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136394    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:47.146148 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136397    4912 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136399    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136403    4912 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136405    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136408    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136411    4912 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136413    4912 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136416    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136419    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136421    4912 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136424    4912 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136427    4912 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136429    4912 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136432    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136434    4912 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136437    4912 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136439    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136442    4912 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136444    4912 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:47.146602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136447    4912 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136449    4912 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136451    4912 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136454    4912 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136457    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136460    4912 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136462    4912 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136465    4912 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136468    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136471    4912 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136473    4912 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136476    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136478    4912 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136481    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136483    4912 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136486    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136488    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136491    4912 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136493    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136496    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:47.147276 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136499    4912 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136502    4912 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136504    4912 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136507    4912 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136509    4912 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136512    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136514    4912 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136518    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136520    4912 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136523    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136527    4912 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136531    4912 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136534    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136536    4912 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136539    4912 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136542    4912 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136545    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136549    4912 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136551    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:47.148089 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136554    4912 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136557    4912 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136559    4912 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136562    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136565    4912 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136567    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136570    4912 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136572    4912 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136575    4912 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136577    4912 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136580    4912 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136583    4912 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136585    4912 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136588    4912 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136590    4912 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136593    4912 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136595    4912 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136597    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136600    4912 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136603    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:47.148587 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136605    4912 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136608    4912 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136610    4912 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.136639    4912 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136779    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136785    4912 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136788    4912 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136791    4912 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136794    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136798    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136803    4912 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136808    4912 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136812    4912 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136815    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136817    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:47.149087 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136820    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136823    4912 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136825    4912 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136828    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136831    4912 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136833    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136836    4912 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136839    4912 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136841    4912 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136844    4912 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136847    4912 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136849    4912 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136851    4912 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136854    4912 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136857    4912 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136859    4912 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136862    4912 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136865    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136868    4912 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136870    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:47.149471 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136873    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136876    4912 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136878    4912 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136881    4912 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136883    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136886    4912 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136888    4912 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136892    4912 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136895    4912 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136898    4912 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136901    4912 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136903    4912 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136905    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136909    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136913    4912 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136916    4912 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136919    4912 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136921    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136924    4912 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:47.149984 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136927    4912 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136929    4912 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136932    4912 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136934    4912 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136937    4912 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136939    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136942    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136944    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136947    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136949    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136952    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136954    4912 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136957    4912 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136959    4912 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136961    4912 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136964    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136966    4912 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136969    4912 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136972    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:47.150465 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136975    4912 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136977    4912 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136980    4912 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136983    4912 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136985    4912 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136988    4912 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136990    4912 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136993    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136995    4912 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.136998    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137000    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137003    4912 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137005    4912 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137008    4912 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137011    4912 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137013    4912 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137016    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137018    4912 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137020    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137023    4912 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:47.150934 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137026    4912 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137028    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137031    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137033    4912 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137036    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137038    4912 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137041    4912 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137044    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137047    4912 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137050    4912 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137069    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137073    4912 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137076    4912 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137079    4912 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137082    4912 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137085    4912 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:47.151576 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137088    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.137111    4912 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137150    4912 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137155    4912 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137158    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137161    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137165    4912 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137168    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137170    4912 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137173    4912 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137175    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137178    4912 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137180    4912 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137183    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137185    4912 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137188    4912 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:47.151967 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137190    4912 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137193    4912 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137195    4912 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137198    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137200    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137203    4912 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137206    4912 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137208    4912 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137211    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137213    4912 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137222    4912 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137226    4912 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137230    4912 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137233    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137236    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137239    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137242    4912 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137244    4912 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137247    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:47.152377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137250    4912 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137253    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137255    4912 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137258    4912 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137260    4912 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137263    4912 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137265    4912 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137268    4912 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137270    4912 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137273    4912 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137276    4912 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137278    4912 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137281    4912 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137283    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137286    4912 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137288    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137291    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137294    4912 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137296    4912 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137299    4912 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:47.152841 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137301    4912 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137304    4912 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137306    4912 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137310    4912 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137313    4912 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137316    4912 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137319    4912 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137321    4912 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137324    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137326    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137331    4912 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137334    4912 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137337    4912 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137339    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137342    4912 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137345    4912 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137347    4912 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137350    4912 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137352    4912 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:47.153361 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137355    4912 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137357    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137360    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137362    4912 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137365    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137368    4912 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137370    4912 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137373    4912 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137375    4912 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137377    4912 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137381    4912 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137384    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137387    4912 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137389    4912 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137392    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137394    4912 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137398    4912 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137401    4912 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137404    4912 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:47.153824 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137406    4912 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137409    4912 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137411    4912 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137414    4912 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137416    4912 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137419    4912 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137422    4912 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137426    4912 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137428    4912 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137431    4912 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137433    4912 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137436    4912 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137438    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137441    4912 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:47.137443    4912 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.137494    4912 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.138517    4912 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.138582    4912 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.138753    4912 server.go:1014] "Starting client certificate rotation"
Nov 02 01:31:47.154324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.138902    4912 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:31:47.154773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.138963    4912 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:26:05 +0000 UTC" deadline="2028-07-15 14:14:22.780086216 +0000 UTC"
Nov 02 01:31:47.154773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.138999    4912 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14916h42m35.641090074s"
Nov 02 01:31:47.154773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.139330    4912 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:31:47.154773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.140451    4912 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:31:47.154773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.142044    4912 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:31:47.154773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.143749    4912 log.go:25] "Validated CRI v1 image API"
Nov 02 01:31:47.154773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.144605    4912 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:31:47.154773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.146626    4912 fs.go:135] Filesystem UUIDs: map[01ab4aaf-b537-4bae-aab8-7a362625c98b:/dev/nvme0n1p3 74d42fd2-0b6b-4e5d-aebc-323443ced95b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 01:31:47.154992 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.146652    4912 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02b5e9f0603c323711292c80ad84433b43389c690fcdb3a6263a61a3a9cc9c3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02b5e9f0603c323711292c80ad84433b43389c690fcdb3a6263a61a3a9cc9c3c/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ee4f82ebcf21ee10a638f51a8b3db2081ed076c0999ad91434839929af869c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ee4f82ebcf21ee10a638f51a8b3db2081ed076c0999ad91434839929af869c8/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1eb0faf71a80468585c038798d61123dfce42682f18aade978e135ca9d2535ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1eb0faf71a80468585c038798d61123dfce42682f18aade978e135ca9d2535ed/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3319a73e68a40d31273879a6d1d180caa0d8bf00badba0da6ef585a818c2508a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3319a73e68a40d31273879a6d1d180caa0d8bf00badba0da6ef585a818c2508a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c7fd19fba799063e1e9060bd7b5717b7291455494a28b9c617952a55e170cab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c7fd19fba799063e1e9060bd7b5717b7291455494a28b9c617952a55e170cab/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d775d12f8252fb7e43a677dc50a3dfcc8672c5f19f26fcb464538f80298c5ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d775d12f8252fb7e43a677dc50a3dfcc8672c5f19f26fcb464538f80298c5ef/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a8dc9f2139340a5111148a200f3e2b4d24896cf51c6fe2b265e612683e14e42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a8dc9f2139340a5111148a200f3e2b4d24896cf51c6fe2b265e612683e14e42/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51705617cf3a2ecf3259819f8edf49e6d6572dff388b38cfcac85423b2b9678c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51705617cf3a2ecf3259819f8edf49e6d6572dff388b38cfcac85423b2b9678c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b08196084d613a5f329f0ff23b4e4ba494a840c36dc11d86da8844fe5f9d720/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b08196084d613a5f329f0ff23b4e4ba494a840c36dc11d86da8844fe5f9d720/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d0c5618fa8f8cdb5d95e8c66bd16de3a9b5ce06e3ec5403f920bc086ab55cc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d0c5618fa8f8cdb5d95e8c66bd16de3a9b5ce06e3ec5403f920bc086ab55cc3/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60d702aa63de9a0aa6b1ca5e905eceb16f2afc3af64eecafbace041b03e876b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60d702aa63de9a0aa6b1ca5e905eceb16f2afc3af64eecafbace041b03e876b1/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cb6b5398a193c429473f822e2aca3b4a96addbdabbf64994373462fff87c691/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cb6b5398a193c429473f822e2aca3b4a96addbdabbf64994373462fff87c691/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/790fcf472d67dced95cbb5e648ef4c6d0d66e731b42699588dd0c0ebf1975410/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/790fcf472d67dced95cbb5e648ef4c6d0d66e731b42699588dd0c0ebf1975410/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ac78d5863a5ee181baa4aa429370961546a266d1891d38161a98b6864b1a6fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ac78d5863a5ee181baa4aa429370961546a266d1891d38161a98b6864b1a6fb/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/805472184fa800d13999646428a1783a4d83d89725ce7c11d5392d12b3d378b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/805472184fa800d13999646428a1783a4d83d89725ce7c11d5392d12b3d378b1/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8292077652b355c56bcdce4f29f614a97ba3380fcf87d41b3a83e1518691a626/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8292077652b355c56bcdce4f29f614a97ba3380fcf87d41b3a83e1518691a626/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cb6fff0d1293c6a6ae781628682e876f986b914f1c4dda9fd3ce8c5643c82a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cb6fff0d1293c6a6ae781628682e876f986b914f1c4dda9fd3ce8c5643c82a1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90e97ab6e2ab415aba3c2edd70f326729b73a49ac6767006a660bb39289edea3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90e97ab6e2ab415aba3c2edd70f326729b73a49ac6767006a660bb39289edea3/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/934ff7aa8060dc5cf7622f64ca2125527dc3448dac4ff74059aaa8838df4694f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/934ff7aa8060dc5cf7622f64ca2125527dc3448dac4ff74059aaa8838df4694f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2a89733b88aea8224676f032630116b71e58a415208c3ff38c23b1668f2bdab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2a89733b88aea8224676f032630116b71e58a415208c3ff38c23b1668f2bdab/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0c7a4d9f829aac640f265a026904daae2465fb7ea4da328207038ef7076aa72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0c7a4d9f829aac640f265a026904daae2465fb7ea4da328207038ef7076aa72/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e26a637c97de91d54768fa1a41f450110d39e6f07b6ba622e7245b6e9bf69178/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e26a637c97de91d54768fa1a41f450110d39e6f07b6ba622e7245b6e9bf69178/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e368bf4a9b539e4b1a3f41970c9677b3ec8fb8d678a615afbbb5a435463eb541/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e368bf4a9b539e4b1a3f41970c9677b3ec8fb8d678a615afbbb5a435463eb541/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08bd96c5-7bd3-4b79-847c-fd7f7866e231/volumes/kubernetes.io~projected/kube-api-access-28lx6:{mountpoint:/var/lib/kubelet/pods/08bd96c5-7bd3-4b79-847c-fd7f7866e231/volumes/kubernetes.io~projected/kube-api-access-28lx6 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b3aaf5a-58ac-4fa6-846e-0764a4803577/volumes/kubernetes.io~projected/kube-api-access-ncsgc:{mountpoint:/var/lib/kubelet/pods/1b3aaf5a-58ac-4fa6-846e-0764a4803577/volumes/kubernetes.io~projected/kube-api-access-ncsgc major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b8b7f85-d24b-4ee9-a036-54554ab354ec/volumes/kubernetes.io~projected/kube-api-access-qr7tt:{mountpoint:/var/lib/kubelet/pods/1b8b7f85-d24b-4ee9-a036-54554ab354ec/volumes/kubernetes.io~projected/kube-api-access-qr7tt major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/230459c8-baa4-47cc-9cc2-73f0535fb650/volumes/kubernetes.io~projected/kube-api-access-r8kn5:{mountpoint:/var/lib/kubelet/pods/230459c8-baa4-47cc-9cc2-73f0535fb650/volumes/kubernetes.io~projected/kube-api-access-r8kn5 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/277a3a65-7e29-4cb7-a913-d364040935b0/volumes/kubernetes.io~projected/kube-api-access-42z6t:{mountpoint:/var/lib/kubelet/pods/277a3a65-7e29-4cb7-a913-d364040935b0/volumes/kubernetes.io~projected/kube-api-access-42z6t major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/277a3a65-7e29-4cb7-a913-d364040935b0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/277a3a65-7e29-4cb7-a913-d364040935b0/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aba37fc-45ec-4861-b481-f022d9727476/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2aba37fc-45ec-4861-b481-f022d9727476/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4161f42e-a6d1-48ea-b5dc-4d21e16facf6/volumes/kubernetes.io~projected/kube-api-access-v4p22:{mountpoint:/var/lib/kubelet/pods/4161f42e-a6d1-48ea-b5dc-4d21e16facf6/volumes/kubernetes.io~projected/kube-api-access-v4p22 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/444113a1-c287-4a62-98ba-7b88fe123732/volumes/kubernetes.io~projected/kube-api-access-x7xqs:{mountpoint:/var/lib/kubelet/pods/444113a1-c287-4a62-98ba-7b88fe123732/volumes/kubernetes.io~projected/kube-api-access-x7xqs major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/445a4994-d804-4369-916f-e9d14143f0c1/volumes/kubernetes.io~projected/kube-api-access-x422d:{mountpoint:/var/lib/kubelet/pods/445a4994-d804-4369-916f-e9d14143f0c1/volumes/kubernetes.io~projected/kube-api-access-x422d major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/447b6716-29a3-4000-8df4-022f92de0e23/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/447b6716-29a3-4000-8df4-022f92de0e23/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48e3609b-a3dc-4890-aab1-e4ecd697f718/volumes/kubernetes.io~projected/kube-api-access-tgnpm:{mountpoint:/var/lib/kubelet/pods/48e3609b-a3dc-4890-aab1-e4ecd697f718/volumes/kubernetes.io~projected/kube-api-access-tgnpm major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d375cf0-b26a-45d5-b6ab-23f752812dd5/volumes/kubernetes.io~projected/kube-api-access-pw9cx:{mountpoint:/var/lib/kubelet/pods/4d375cf0-b26a-45d5-b6ab-23f752812dd5/volumes/kubernetes.io~projected/kube-api-access-pw9cx major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5826852c-0e9b-4031-8213-94051d31c42c/volumes/kubernetes.io~projected/kube-api-access-fjqp2:{mountpoint:/var/lib/kubelet/pods/5826852c-0e9b-4031-8213-94051d31c42c/volumes/kubernetes.io~projected/kube-api-access-fjqp2 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5826852c-0e9b-4031-8213-94051d31c42c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/5826852c-0e9b-4031-8213-94051d31c42c/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5826852c-0e9b-4031-8213-94051d31c42c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/5826852c-0e9b-4031-8213-94051d31c42c/volumes/kubernetes.io~secret/stats-auth major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c414b43-ca48-4f3e-a38a-c82f2cc27283/volumes/kubernetes.io~projected/kube-api-access-wq5dl:{mountpoint:/var/lib/kubelet/pods/5c414b43-ca48-4f3e-a38a-c82f2cc27283/volumes/kubernetes.io~projected/kube-api-access-wq5dl major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c414b43-ca48-4f3e-a38a-c82f2cc27283/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5c414b43-ca48-4f3e-a38a-c82f2cc27283/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/volumes/kubernetes.io~projected/kube-api-access-58kn4:{mountpoint:/var/lib/kubelet/pods/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/volumes/kubernetes.io~projected/kube-api-access-58kn4 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~projected/kube-api-access-x7ppz:{mountpoint:/var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~projected/kube-api-access-x7ppz major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c3d1051-a370-4a67-bebf-78d6cd902101/volumes/kubernetes.io~projected/kube-api-access-lqgxf:{mountpoint:/var/lib/kubelet/pods/6c3d1051-a370-4a67-bebf-78d6cd902101/volumes/kubernetes.io~projected/kube-api-access-lqgxf major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7121a072-9bc5-41fb-920e-bff50c7cd95f/volumes/kubernetes.io~projected/kube-api-access-ztztb:{mountpoint:/var/lib/kubelet/pods/7121a072-9bc5-41fb-920e-bff50c7cd95f/volumes/kubernetes.io~projected/kube-api-access-ztztb major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77d4f2e6-e448-4f1e-b817-50cb130f99c1/volumes/kubernetes.io~projected/kube-api-access-gfl22:{mountpoint:/var/lib/kubelet/pods/77d4f2e6-e448-4f1e-b817-50cb130f99c1/volumes/kubernetes.io~projected/kube-api-access-gfl22 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77d4f2e6-e448-4f1e-b817-50cb130f99c1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/77d4f2e6-e448-4f1e-b817-50cb130f99c1/volumes/kubernetes.io~secret/cert major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4/volumes/kubernetes.io~projected/kube-api-access-cnvdl:{mountpoint:/var/lib/kubelet/pods/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4/volumes/kubernetes.io~projected/kube-api-access-cnvdl major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~projected/kube-api-access-jjcgz:{mountpoint:/var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~projected/kube-api-access-jjcgz major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/944b4b8f-a639-4153-a4f7-3dc0e5691225/volumes/kubernetes.io~projected/kube-api-access-5hlm9:{mountpoint:/var/lib/kubelet/pods/944b4b8f-a639-4153-a4f7-3dc0e5691225/volumes/kubernetes.io~projected/kube-api-access-5hlm9 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98ad998e-3ed5-4892-b6d1-f14e516be75a/volumes/kubernetes.io~projected/kube-api-access-25zwf:{mountpoint:/var/lib/kubelet/pods/98ad998e-3ed5-4892-b6d1-f14e516be75a/volumes/kubernetes.io~projected/kube-api-access-25zwf major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9e45b19-94de-40fc-81c4-960324b99e20/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a9e45b19-94de-40fc-81c4-960324b99e20/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9e45b19-94de-40fc-81c4-960324b99e20/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a9e45b19-94de-40fc-81c4-960324b99e20/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9e45b19-94de-40fc-81c4-960324b99e20/volumes/kubernetes.io~projected/kube-api-access-tpzsd:{mountpoint:/var/lib/kubelet/pods/a9e45b19-94de-40fc-81c4-960324b99e20/volumes/kubernetes.io~projected/kube-api-access-tpzsd major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adda0d83-929a-4e90-a939-49ba1f7f73b4/volumes/kubernetes.io~projected/kube-api-access-tstcv:{mountpoint:/var/lib/kubelet/pods/adda0d83-929a-4e90-a939-49ba1f7f73b4/volumes/kubernetes.io~projected/kube-api-access-tstcv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1284f23-412e-4a65-970e-25ec56f3da4a/volumes/kubernetes.io~projected/kube-api-access-6b9vc:{mountpoint:/var/lib/kubelet/pods/c1284f23-412e-4a65-970e-25ec56f3da4a/volumes/kubernetes.io~projected/kube-api-access-6b9vc major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1284f23-412e-4a65-970e-25ec56f3da4a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c1284f23-412e-4a65-970e-25ec56f3da4a/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce9b8562-0baf-437e-bf85-5cc8f0e32de1/volumes/kubernetes.io~projected/kube-api-access-7tgxv:{mountpoint:/var/lib/kubelet/pods/ce9b8562-0baf-437e-bf85-5cc8f0e32de1/volumes/kubernetes.io~projected/kube-api-access-7tgxv major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3c0e9d9-eda0-4e8f-8677-0bd4406243a4/volumes/kubernetes.io~projected/kube-api-access-4mxgq:{mountpoint:/var/lib/kubelet/pods/f3c0e9d9-eda0-4e8f-8677-0bd4406243a4/volumes/kubernetes.io~projected/kube-api-access-4mxgq major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f74b04ce-df4a-46d3-9eb5-52464d2b5e50/volumes/kubernetes.io~projected/kube-api-access-jt47v:{mountpoint:/var/lib/kubelet/pods/f74b04ce-df4a-46d3-9eb5-52464d2b5e50/volumes/kubernetes.io~projected/kube-api-access-jt47v major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbde9de9-c649-40ae-a43e-896097fc7dcd/volumes/kubernetes.io~projected/kube-api-access-z69bj:{mountpoint:/var/lib/kubelet/pods/fbde9de9-c649-40ae-a43e-896097fc7dcd/volumes/kubernetes.io~projected/kube-api-access-z69bj major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1f8ba1cd8b98646cd7a209115b2b84820823d7777c112709f44c48005570a17b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d14125bb053322551eca44f71648a63eba2f5be928d5519e8deb22589226eab4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/de6b6978f924008646438c8d8423a048dc6287bda85d207f44543ea77df2b281/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/10f7ade1dcfda35050c4ccc28e6d261efe92593adf85f7cbda4fe361923b132b/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cb3b0c2d8156de1c3c61dea41a412ffe4b177b39bcc188f9d5d27ca3ba4ec583/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6633e35bd4d791a0329a095602b76a97cb1090c1b49e52aa7d6baaf3a2453e4a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fe3fc61fabbc163cfc4bdad66545235163f2e408c47fd271783a28332db6fc10/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/eeab0e5dde527c0baa528f31ef664f96aa3baf33df695c5afbf0e4ef46d36ee1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1e0b8e90d9e3d90aa6a388e5e9b0936a8622bd83a52854f76b4cab03ab81d6b5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/50d2eb26edf2af0d8a1a155ba931a977681910a195ba5a3ad4f08a9ef5071b38/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/6e3a757c176864c0e12c73303b666503d0c2b03972e66417c10aed8b0e867fcb/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/b8f42a61dedfb0a0366dfabe37ae0569ec0c81651e7f9de5b1c8dda9982ca663/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/885ceaa8f67e13eb2e1c452f8d3cdf487fab4d4b3554bc999279893d9c9070fc/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/aeb72362f44623bc597750eedb3cf9cdea649ff5f9494b2f51cff18b2e0cc450/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b42a7578228e09e469c8ab8a884be50e11f87691754175751456ae0f60be5afa/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6edff4216753d363406a72e44159181266901ffe062c1b26fad512ab4bb9fbc1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a9822f491134fb10e59232eb1c48e0679dd8c72c44437ed6ed14f87e3e5d27eb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fe5096d0629e0b217893469e697e572b35210fb65f86b467c29d9610a70321b3/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4b7e1e4c6acd27d1e9625be678dd130db7ca76e220ecdb11fac9783897a87f2b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f73043036fadc22d403311f48bbbc99be0111a521675f1a62fb640971a0344d0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/58675be40d608f15f90bfc5aedb2d41d1504ac1b3f8e84f2d47b03028a083fe0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/a516b12747baaeb55995c594476cc33c751b934101413331c1a93127134bcef6/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/530fa7756928305003ff6b4b6691949e92960ec3b40e79a1f0bcfc8c73191687/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/24d7ff6ee1c2884a9c7e7bc321543afb494e720d9694cba873c18b3fe457ba18/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/05fcb66ce9cb5dcc108cde46e842dd5f4ca9e83a7799a235b8be82088f97f6e1/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/3430391384d115b1db95e9ae0e7f9d78e7068d2b9defe9c95326f202f9fd718e/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/023eda84c6da120b46b1f459c2ca103ac83776a6a0fb516378cc3328b61c0c79/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/88b25cb238bda547fe2b4e6fe7538d10aafa838eb1b9c7efd8fc0cbb17f80b8d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/22e22f81589c607fead852144090818174eada004c5bd18c1829137936193fe6/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/1bd5b8a7c274e0e2103ce7868050ada958fe785a9ce2f674743fc9f19c63794b/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/aa951f998eb1eccd8ac97acc2aa4b24e2cac680a1c9dbfb54ab4993b2fb3f40c/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/8335f18e7f997dd2c06478f8a41250bf40e0f1b23e192df5dd4e07ebb419fef0/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/d741e3612bd7e7c2e405244129e992f913087971f81ba9d881f879b15a9ba962/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/2141f1661a9d66ccd0dd72cdc0a8fc7b856524716200aea2228e36abb5a1219e/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/7f29855596a25bc36e785bf1b043e364dc3aadaf974832eae16d7d46924a2d33/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/a7bdf9bbc47bd0942dc24ef5b6dbd6b0cf5175fcc23ef74c689176b4fdd1f871/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/9e50361dcf5339a06a42e1f81e5c1063f1e105e6bdec9494b59c22204eac7cd6/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/11a0e0fae7cabb70a7d55f3bf285d8db4e23b00822d1aeefcc552ae68fb01fd1/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/63cabc06f98a92396105029e4989ee316b3dd2c433c4ae5a2859e8a2caf5ca22/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/fd66dd2b798ceaf69842bab46f859fc3d8abf61339be1ee8f2ab136fe72c2ed8/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/1eec772bcf106c90dc3e310ae51acf3f60b155555bd723153f907fefaba80b2a/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/c8bfd3325c5d3b4de8dc1ad89bf04181e255112b1d68ebfd598582094cab4160/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/bdc55f48f66d3c1659f96e77e0d0766d1b52e4d37b47ca9e610efb92f07eca1e/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/f632a9b9d7543931bb453c03ac10e20e28e5337149d0f13bf57ab41341f3e0e4/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/08a8c5d7ac265ad180d242cd1bb4e39b82a008db45da5ef45283515cffff458a/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/6c255522231e7fea210444cb55852a23c309f021287d35ecfcc973165de15fc2/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/f0109682c4532750899b8352a6f1496fb7eaf5503b43f9788c7f9f4a45ae9193/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5fa4a40a6894cb4900debe88f200e954ea50f6c98f4c0400e01b8e9a55e0c76b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/cd0e3febe86bd1bb8a40e8e0bcf0db093ceb1797dff7478390d706a5e90f8b0d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d6f4ebde1994fb3eba6e9af1f94bcbd34706e1f10a9d91e6f1f4f7319f8d4488/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/1deaff7bc4b8afbd992c9619e4ce40fabe2fe61c078098b1c260a976cbe3c014/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6161259632e70407d51a37b434288ea7d49701086a891b99bac65c22464c0b32/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ed8f186ffb28020ff0231e9c0eb920a22e4f0fd2793718cd82b82dfec3eaf40d/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/aa014b2999bb95dc694a85fc61f933c0b8e75be3caca7e6d1443749478d99bc4/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/149e2608bd693f56cf547f9cfe6d3661ae8ee1ed9cb83610144f0a44ded392e1/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 01:31:47.157994 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.157660    4912 manager.go:217] Machine: {Timestamp:2026-11-02 01:31:47.157181883 +0000 UTC m=+0.062647240 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100686 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2924a6757e535303c5c877aaa3a830 SystemUUID:ec2924a6-757e-5353-03c5-c877aaa3a830 BootID:3c7b2f71-06d1-42ef-a453-3db537f35bf3 Filesystems:[{Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/444113a1-c287-4a62-98ba-7b88fe123732/volumes/kubernetes.io~projected/kube-api-access-x7xqs DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e26a637c97de91d54768fa1a41f450110d39e6f07b6ba622e7245b6e9bf69178/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0c7a4d9f829aac640f265a026904daae2465fb7ea4da328207038ef7076aa72/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3319a73e68a40d31273879a6d1d180caa0d8bf00badba0da6ef585a818c2508a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cb6fff0d1293c6a6ae781628682e876f986b914f1c4dda9fd3ce8c5643c82a1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5826852c-0e9b-4031-8213-94051d31c42c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b08196084d613a5f329f0ff23b4e4ba494a840c36dc11d86da8844fe5f9d720/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b3aaf5a-58ac-4fa6-846e-0764a4803577/volumes/kubernetes.io~projected/kube-api-access-ncsgc DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/805472184fa800d13999646428a1783a4d83d89725ce7c11d5392d12b3d378b1/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a9e45b19-94de-40fc-81c4-960324b99e20/volumes/kubernetes.io~projected/kube-api-access-tpzsd DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90e97ab6e2ab415aba3c2edd70f326729b73a49ac6767006a660bb39289edea3/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7121a072-9bc5-41fb-920e-bff50c7cd95f/volumes/kubernetes.io~projected/kube-api-access-ztztb DeviceMajor:0 DeviceMinor:314 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d0c5618fa8f8cdb5d95e8c66bd16de3a9b5ce06e3ec5403f920bc086ab55cc3/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48e3609b-a3dc-4890-aab1-e4ecd697f718/volumes/kubernetes.io~projected/kube-api-access-tgnpm DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c3d1051-a370-4a67-bebf-78d6cd902101/volumes/kubernetes.io~projected/kube-api-access-lqgxf DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5826852c-0e9b-4031-8213-94051d31c42c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1eb0faf71a80468585c038798d61123dfce42682f18aade978e135ca9d2535ed/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/adda0d83-929a-4e90-a939-49ba1f7f73b4/volumes/kubernetes.io~projected/kube-api-access-tstcv DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2aba37fc-45ec-4861-b481-f022d9727476/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f74b04ce-df4a-46d3-9eb5-52464d2b5e50/volumes/kubernetes.io~projected/kube-api-access-jt47v DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3c0e9d9-eda0-4e8f-8677-0bd4406243a4/volumes/kubernetes.io~projected/kube-api-access-4mxgq DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~projected/kube-api-access-jjcgz DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e368bf4a9b539e4b1a3f41970c9677b3ec8fb8d678a615afbbb5a435463eb541/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a8dc9f2139340a5111148a200f3e2b4d24896cf51c6fe2b265e612683e14e42/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c1284f23-412e-4a65-970e-25ec56f3da4a/volumes/kubernetes.io~projected/kube-api-access-6b9vc DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5c414b43-ca48-4f3e-a38a-c82f2cc27283/volumes/kubernetes.io~projected/kube-api-access-wq5dl DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5826852c-0e9b-4031-8213-94051d31c42c/volumes/kubernetes.io~projected/kube-api-access-fjqp2 DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/934ff7aa8060dc5cf7622f64ca2125527dc3448dac4ff74059aaa8838df4694f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/230459c8-baa4-47cc-9cc2-73f0535fb650/volumes/kubernetes.io~projected/kube-api-access-r8kn5 DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b8b7f85-d24b-4ee9-a036-54554ab354ec/volumes/kubernetes.io~projected/kube-api-access-qr7tt DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4/volumes/kubernetes.io~projected/kube-api-access-cnvdl DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ee4f82ebcf21ee10a638f51a8b3db2081ed076c0999ad91434839929af869c8/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c414b43-ca48-4f3e-a38a-c82f2cc27283/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/volumes/kubernetes.io~projected/kube-api-access-58kn4 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~projected/kube-api-access-x7ppz DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/445a4994-d804-4369-916f-e9d14143f0c1/volumes/kubernetes.io~projected/kube-api-access-x422d DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60d702aa63de9a0aa6b1ca5e905eceb16f2afc3af64eecafbace041b03e876b1/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/277a3a65-7e29-4cb7-a913-d364040935b0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/77d4f2e6-e448-4f1e-b817-50cb130f99c1/volumes/kubernetes.io~projected/kube-api-access-gfl22 DeviceMajor:0 DeviceMinor:241 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/08bd96c5-7bd3-4b79-847c-fd7f7866e231/volumes/kubernetes.io~projected/kube-api-access-28lx6 DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a9e45b19-94de-40fc-81c4-960324b99e20/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/790fcf472d67dced95cbb5e648ef4c6d0d66e731b42699588dd0c0ebf1975410/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ce9b8562-0baf-437e-bf85-5cc8f0e32de1/volumes/kubernetes.io~projected/kube-api-access-7tgxv DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c7fd19fba799063e1e9060bd7b5717b7291455494a28b9c617952a55e170cab/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98ad998e-3ed5-4892-b6d1-f14e516be75a/volumes/kubernetes.io~projected/kube-api-access-25zwf DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1284f23-412e-4a65-970e-25ec56f3da4a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d375cf0-b26a-45d5-b6ab-23f752812dd5/volumes/kubernetes.io~projected/kube-api-access-pw9cx DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fbde9de9-c649-40ae-a43e-896097fc7dcd/volumes/kubernetes.io~projected/kube-api-access-z69bj DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cb6b5398a193c429473f822e2aca3b4a96addbdabbf64994373462fff87c691/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/277a3a65-7e29-4cb7-a913-d364040935b0/volumes/kubernetes.io~projected/kube-api-access-42z6t DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/944b4b8f-a639-4153-a4f7-3dc0e5691225/volumes/kubernetes.io~projected/kube-api-access-5hlm9 DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d775d12f8252fb7e43a677dc50a3dfcc8672c5f19f26fcb464538f80298c5ef/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ac78d5863a5ee181baa4aa429370961546a266d1891d38161a98b6864b1a6fb/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/447b6716-29a3-4000-8df4-022f92de0e23/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a9e45b19-94de-40fc-81c4-960324b99e20/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51705617cf3a2ecf3259819f8edf49e6d6572dff388b38cfcac85423b2b9678c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77d4f2e6-e448-4f1e-b817-50cb130f99c1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:239 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2a89733b88aea8224676f032630116b71e58a415208c3ff38c23b1668f2bdab/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8292077652b355c56bcdce4f29f614a97ba3380fcf87d41b3a83e1518691a626/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4161f42e-a6d1-48ea-b5dc-4d21e16facf6/volumes/kubernetes.io~projected/kube-api-access-v4p22 DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02b5e9f0603c323711292c80ad84433b43389c690fcdb3a6263a61a3a9cc9c3c/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02b5e9f0603c323 MacAddress:f2:34:8b:d4:51:12 Speed:10000 Mtu:8901} {Name:0ee4f82ebcf21ee MacAddress:0e:3e:68:73:e4:c2 Speed:10000 Mtu:8901} {Name:3c7fd19fba79906 MacAddress:3a:12:6d:49:c2:89 Speed:10000 Mtu:8901} {Name:4a8dc9f2139340a MacAddress:ae:a7:91:77:3f:1f Speed:10000 Mtu:8901} {Name:5d0c5618fa8f8cd MacAddress:0e:f3:f3:9c:3e:fd Speed:10000 Mtu:8901} {Name:60d702aa63de9a0 MacAddress:8e:06:96:b2:66:bb Speed:10000 Mtu:8901} {Name:805472184fa800d MacAddress:2e:31:db:14:9d:95 Speed:10000 Mtu:8901} {Name:8292077652b355c MacAddress:b6:3f:4a:88:36:c9 Speed:10000 Mtu:8901} {Name:90e97ab6e2ab415 MacAddress:ce:64:91:9a:30:a2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:56:d9:ed:5e:19 Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:8a:96:4a:04:17 Speed:0 Mtu:8901} {Name:d2a89733b88aea8 MacAddress:76:ee:71:f5:2c:0c Speed:10000 Mtu:8901} {Name:e0c7a4d9f829aac MacAddress:16:5f:4f:ac:cc:42 Speed:10000 Mtu:8901} {Name:e26a637c97de91d MacAddress:2e:da:68:a8:d0:0f Speed:10000 Mtu:8901} {Name:e368bf4a9b539e4 MacAddress:02:6b:ab:21:cf:fe Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:56:d9:ed:5e:19 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e2:5f:f4:58:a2:e4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:89:ec:b0:b2:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:31:47.158222 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.157998    4912 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:31:47.158222 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.158044    4912 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:31:47.158459 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.158436    4912 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:31:47.158592 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.158460    4912 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-33.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:31:47.158634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.158594    4912 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:31:47.158634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.158602    4912 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:31:47.158634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.158613    4912 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:31:47.158634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.158632    4912 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:31:47.158742 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.158712    4912 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:31:47.158912 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.158902    4912 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:31:47.159040 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.159029    4912 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:31:47.159103 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.159094    4912 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:31:47.159140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.159108    4912 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:31:47.159140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.159119    4912 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:31:47.159140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.159128    4912 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:31:47.159140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.159136    4912 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:31:47.160204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.160186    4912 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:31:47.160204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.160202    4912 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:31:47.160903 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.160870    4912 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:31:47.173906 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.173885    4912 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:31:47.173906 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.173903    4912 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:31:47.174110 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174021    4912 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 01:31:47.174294 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174278    4912 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:31:47.174368 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174319    4912 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:31:47.174451 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174440    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:31:47.174506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174460    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:31:47.174506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174471    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:31:47.174506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174480    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:31:47.174506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174490    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:31:47.174506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174499    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:31:47.174738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174510    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:31:47.174738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174519    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:31:47.174738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174530    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:31:47.174738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174541    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:31:47.174738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174556    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:31:47.174738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174571    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:31:47.174738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174605    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:31:47.174738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.174614    4912 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:31:47.178317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.178300    4912 server.go:1263] "Started kubelet"
Nov 02 01:31:47.178416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.178371    4912 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:31:47.179615 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.179470    4912 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:31:47.179615 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.179587    4912 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:31:47.179622 ip-10-0-135-33 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:31:47.182615 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.181382    4912 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:31:47.183338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.183318    4912 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.184175    4912 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.184200    4912 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.184247    4912 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:26:06 +0000 UTC" deadline="2028-04-02 03:02:46.854484503 +0000 UTC"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.184265    4912 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12409h30m59.670222489s"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.184295    4912 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.184302    4912 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.184372    4912 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.184878    4912 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-33.us-west-1.compute.internal\" not found"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.185440    4912 factory.go:55] Registering systemd factory
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.185458    4912 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.185872    4912 factory.go:153] Registering CRI-O factory
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.185886    4912 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.185958    4912 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.185978    4912 factory.go:103] Registering Raw factory
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.185998    4912 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.186529    4912 manager.go:319] Starting recovery of all containers
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187599    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="277a3a65-7e29-4cb7-a913-d364040935b0" volumeName="kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-trusted-ca" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187628    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="277a3a65-7e29-4cb7-a913-d364040935b0" volumeName="kubernetes.io/secret/277a3a65-7e29-4cb7-a913-d364040935b0-serving-cert" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187657    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4161f42e-a6d1-48ea-b5dc-4d21e16facf6" volumeName="kubernetes.io/projected/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-kube-api-access-v4p22" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187674    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="445a4994-d804-4369-916f-e9d14143f0c1" volumeName="kubernetes.io/projected/445a4994-d804-4369-916f-e9d14143f0c1-kube-api-access-x422d" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187685    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" volumeName="kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187711    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08bd96c5-7bd3-4b79-847c-fd7f7866e231" volumeName="kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187724    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="447b6716-29a3-4000-8df4-022f92de0e23" volumeName="kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187735    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5826852c-0e9b-4031-8213-94051d31c42c" volumeName="kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-stats-auth" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187766    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77d4f2e6-e448-4f1e-b817-50cb130f99c1" volumeName="kubernetes.io/secret/77d4f2e6-e448-4f1e-b817-50cb130f99c1-cert" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187779    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9404b1b3-624d-4246-a20a-8627cfd58e2c" volumeName="kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187791    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9404b1b3-624d-4246-a20a-8627cfd58e2c" volumeName="kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187818    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce" volumeName="kubernetes.io/projected/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-kube-api-access-58kn4" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187835    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" volumeName="kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187848    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9404b1b3-624d-4246-a20a-8627cfd58e2c" volumeName="kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187906    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98ad998e-3ed5-4892-b6d1-f14e516be75a" volumeName="kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187918    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce9b8562-0baf-437e-bf85-5cc8f0e32de1" volumeName="kubernetes.io/projected/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-kube-api-access-7tgxv" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187929    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b8b7f85-d24b-4ee9-a036-54554ab354ec" volumeName="kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187956    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4161f42e-a6d1-48ea-b5dc-4d21e16facf6" volumeName="kubernetes.io/empty-dir/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-tmp-dir" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187968    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="444113a1-c287-4a62-98ba-7b88fe123732" volumeName="kubernetes.io/projected/444113a1-c287-4a62-98ba-7b88fe123732-kube-api-access-x7xqs" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.187979    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1284f23-412e-4a65-970e-25ec56f3da4a" volumeName="kubernetes.io/projected/c1284f23-412e-4a65-970e-25ec56f3da4a-kube-api-access-6b9vc" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188003    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aba37fc-45ec-4861-b481-f022d9727476" volumeName="kubernetes.io/configmap/2aba37fc-45ec-4861-b481-f022d9727476-konnectivity-ca" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188015    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48e3609b-a3dc-4890-aab1-e4ecd697f718" volumeName="kubernetes.io/configmap/48e3609b-a3dc-4890-aab1-e4ecd697f718-serviceca" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188041    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944b4b8f-a639-4153-a4f7-3dc0e5691225" volumeName="kubernetes.io/projected/944b4b8f-a639-4153-a4f7-3dc0e5691225-kube-api-access-5hlm9" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188086    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5826852c-0e9b-4031-8213-94051d31c42c" volumeName="kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-default-certificate" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188099    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="277a3a65-7e29-4cb7-a913-d364040935b0" volumeName="kubernetes.io/projected/277a3a65-7e29-4cb7-a913-d364040935b0-kube-api-access-42z6t" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188114    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4161f42e-a6d1-48ea-b5dc-4d21e16facf6" volumeName="kubernetes.io/configmap/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-config-volume" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188141    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98ad998e-3ed5-4892-b6d1-f14e516be75a" volumeName="kubernetes.io/projected/98ad998e-3ed5-4892-b6d1-f14e516be75a-kube-api-access-25zwf" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188153    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9e45b19-94de-40fc-81c4-960324b99e20" volumeName="kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-etc-tuned" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188169    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b8b7f85-d24b-4ee9-a036-54554ab354ec" volumeName="kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188183    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4" volumeName="kubernetes.io/projected/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-kube-api-access-cnvdl" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188207    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b8b7f85-d24b-4ee9-a036-54554ab354ec" volumeName="kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188219    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce" volumeName="kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188232    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77d4f2e6-e448-4f1e-b817-50cb130f99c1" volumeName="kubernetes.io/projected/77d4f2e6-e448-4f1e-b817-50cb130f99c1-kube-api-access-gfl22" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188244    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="230459c8-baa4-47cc-9cc2-73f0535fb650" volumeName="kubernetes.io/projected/230459c8-baa4-47cc-9cc2-73f0535fb650-kube-api-access-r8kn5" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188255    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce" volumeName="kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-env-overrides" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188283    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" volumeName="kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188297    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c3d1051-a370-4a67-bebf-78d6cd902101" volumeName="kubernetes.io/projected/6c3d1051-a370-4a67-bebf-78d6cd902101-kube-api-access-lqgxf" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188308    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9404b1b3-624d-4246-a20a-8627cfd58e2c" volumeName="kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188319    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9e45b19-94de-40fc-81c4-960324b99e20" volumeName="kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-tmp" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188330    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1284f23-412e-4a65-970e-25ec56f3da4a" volumeName="kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-service-ca-bundle" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188344    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74b04ce-df4a-46d3-9eb5-52464d2b5e50" volumeName="kubernetes.io/projected/f74b04ce-df4a-46d3-9eb5-52464d2b5e50-kube-api-access-jt47v" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188358    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c414b43-ca48-4f3e-a38a-c82f2cc27283" volumeName="kubernetes.io/configmap/5c414b43-ca48-4f3e-a38a-c82f2cc27283-config" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188372    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c414b43-ca48-4f3e-a38a-c82f2cc27283" volumeName="kubernetes.io/projected/5c414b43-ca48-4f3e-a38a-c82f2cc27283-kube-api-access-wq5dl" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188386    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce" volumeName="kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-config" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188398    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4" volumeName="kubernetes.io/configmap/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-config" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188412    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1284f23-412e-4a65-970e-25ec56f3da4a" volumeName="kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-tmp" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188430    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbde9de9-c649-40ae-a43e-896097fc7dcd" volumeName="kubernetes.io/projected/fbde9de9-c649-40ae-a43e-896097fc7dcd-kube-api-access-z69bj" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188443    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aba37fc-45ec-4861-b481-f022d9727476" volumeName="kubernetes.io/secret/2aba37fc-45ec-4861-b481-f022d9727476-agent-certs" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188457    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" volumeName="kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188469    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98ad998e-3ed5-4892-b6d1-f14e516be75a" volumeName="kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-daemon-config" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188482    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b8b7f85-d24b-4ee9-a036-54554ab354ec" volumeName="kubernetes.io/projected/1b8b7f85-d24b-4ee9-a036-54554ab354ec-kube-api-access-qr7tt" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188496    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26929d97-925d-4839-9bd7-e0fc4b3373a0" volumeName="kubernetes.io/configmap/26929d97-925d-4839-9bd7-e0fc4b3373a0-nginx-conf" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188508    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" volumeName="kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188522    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" volumeName="kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188551    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7121a072-9bc5-41fb-920e-bff50c7cd95f" volumeName="kubernetes.io/projected/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-api-access-ztztb" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188563    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9404b1b3-624d-4246-a20a-8627cfd58e2c" volumeName="kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188586    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9404b1b3-624d-4246-a20a-8627cfd58e2c" volumeName="kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188599    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944b4b8f-a639-4153-a4f7-3dc0e5691225" volumeName="kubernetes.io/configmap/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-config" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188611    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b3aaf5a-58ac-4fa6-846e-0764a4803577" volumeName="kubernetes.io/projected/1b3aaf5a-58ac-4fa6-846e-0764a4803577-kube-api-access-ncsgc" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188625    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5826852c-0e9b-4031-8213-94051d31c42c" volumeName="kubernetes.io/projected/5826852c-0e9b-4031-8213-94051d31c42c-kube-api-access-fjqp2" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188651    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce" volumeName="kubernetes.io/secret/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188666    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7121a072-9bc5-41fb-920e-bff50c7cd95f" volumeName="kubernetes.io/empty-dir/7121a072-9bc5-41fb-920e-bff50c7cd95f-data-volume" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188679    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4" volumeName="kubernetes.io/secret/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-serving-cert" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188691    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adda0d83-929a-4e90-a939-49ba1f7f73b4" volumeName="kubernetes.io/projected/adda0d83-929a-4e90-a939-49ba1f7f73b4-kube-api-access-tstcv" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188703    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1284f23-412e-4a65-970e-25ec56f3da4a" volumeName="kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-snapshots" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188718    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce9b8562-0baf-437e-bf85-5cc8f0e32de1" volumeName="kubernetes.io/configmap/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-telemetry-config" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188731    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="277a3a65-7e29-4cb7-a913-d364040935b0" volumeName="kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-config" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188743    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="445a4994-d804-4369-916f-e9d14143f0c1" volumeName="kubernetes.io/empty-dir/445a4994-d804-4369-916f-e9d14143f0c1-tmp-dir" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188756    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d375cf0-b26a-45d5-b6ab-23f752812dd5" volumeName="kubernetes.io/projected/4d375cf0-b26a-45d5-b6ab-23f752812dd5-kube-api-access-pw9cx" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188769    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" volumeName="kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188781    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9404b1b3-624d-4246-a20a-8627cfd58e2c" volumeName="kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188798    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48e3609b-a3dc-4890-aab1-e4ecd697f718" volumeName="kubernetes.io/projected/48e3609b-a3dc-4890-aab1-e4ecd697f718-kube-api-access-tgnpm" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188822    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c414b43-ca48-4f3e-a38a-c82f2cc27283" volumeName="kubernetes.io/secret/5c414b43-ca48-4f3e-a38a-c82f2cc27283-serving-cert" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188836    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9e45b19-94de-40fc-81c4-960324b99e20" volumeName="kubernetes.io/projected/a9e45b19-94de-40fc-81c4-960324b99e20-kube-api-access-tpzsd" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188850    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1284f23-412e-4a65-970e-25ec56f3da4a" volumeName="kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188863    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1284f23-412e-4a65-970e-25ec56f3da4a" volumeName="kubernetes.io/secret/c1284f23-412e-4a65-970e-25ec56f3da4a-serving-cert" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188875    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c0e9d9-eda0-4e8f-8677-0bd4406243a4" volumeName="kubernetes.io/projected/f3c0e9d9-eda0-4e8f-8677-0bd4406243a4-kube-api-access-4mxgq" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188888    4912 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbde9de9-c649-40ae-a43e-896097fc7dcd" volumeName="kubernetes.io/configmap/fbde9de9-c649-40ae-a43e-896097fc7dcd-iptables-alerter-script" seLinuxMountContext=""
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188902    4912 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:31:47.189246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.188910    4912 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:31:47.193232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.193212    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/1.log"
Nov 02 01:31:47.195729 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.195156    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/1.log"
Nov 02 01:31:47.203220 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.203198    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:31:47.205226 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.205176    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:31:47.215662 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.215610    4912 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:31:47.217113 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.217094    4912 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:31:47.217113 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.217116    4912 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:31:47.217264 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.217142    4912 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:31:47.217264 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.217186    4912 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 01:31:47.227663 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.227643    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/1.log"
Nov 02 01:31:47.227766 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.227669    4912 generic.go:356] "Generic (PLEG): container finished" podID="277a3a65-7e29-4cb7-a913-d364040935b0" containerID="0b2f329e986fdd8bdd7308eed4623b842d3585f482c0e1cf35ea45cc222a453f" exitCode=255
Nov 02 01:31:47.248708 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.248681    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:31:47.249436 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.249402    4912 generic.go:356] "Generic (PLEG): container finished" podID="5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce" containerID="16d22453786eef61b8dd76c46746af591321cb86d3e4c77e9cbea7deb8251035" exitCode=1
Nov 02 01:31:47.256646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.256623    4912 generic.go:356] "Generic (PLEG): container finished" podID="95fc15b85861deeb88c7e9a1573ac9e6" containerID="a90397d2c85a500c6b08b085d31066513f68c9457082b69ca36ed98a88caea6c" exitCode=0
Nov 02 01:31:47.263507 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.263486    4912 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="e5dbe11c75f577477a8e15839e430185b62110d018ef410ae27d882b754cd937" exitCode=0
Nov 02 01:31:47.263507 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.263507    4912 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="ba24fd59729e757849291b4423534833b33b1b3f04884f8f6d2a7a3ee46501e6" exitCode=0
Nov 02 01:31:47.263650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.263513    4912 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="45c6df80488d4d59bf4dfae174ecc73f73786d09b19560e449772ceec546ed07" exitCode=0
Nov 02 01:31:47.263650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.263518    4912 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="a0f3aa432d8427f3add5ed43997709d0fb1c6e1124082122f7e2595a4a35bd66" exitCode=0
Nov 02 01:31:47.263650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.263525    4912 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="66887026d9c3a83aab9d9df27453f3bbab0320828733379070ce9a6de573c109" exitCode=0
Nov 02 01:31:47.263650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.263530    4912 generic.go:356] "Generic (PLEG): container finished" podID="1b8b7f85-d24b-4ee9-a036-54554ab354ec" containerID="6a05dfd8de359cbd78f44720b357d757838bff8aa659dea85d81676201d44735" exitCode=0
Nov 02 01:31:47.274456 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.274440    4912 manager.go:324] Recovery completed
Nov 02 01:31:47.285932 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.285904    4912 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-33.us-west-1.compute.internal\" not found"
Nov 02 01:31:47.293948 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.293933    4912 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:47.295519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.295504    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:47.295581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.295535    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:47.295581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.295545    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:47.298002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.297985    4912 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:31:47.298002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.297999    4912 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:31:47.298002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298003    4912 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:47.298315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298019    4912 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:31:47.298315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298193    4912 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 01:31:47.298315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298209    4912 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 01:31:47.298315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298235    4912 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 01:31:47.298315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298243    4912 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 01:31:47.298315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298252    4912 policy_none.go:50] "Start"
Nov 02 01:31:47.298315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298264    4912 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:31:47.298315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298277    4912 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:31:47.298667 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298460    4912 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 01:31:47.298667 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298470    4912 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 01:31:47.298667 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.298479    4912 policy_none.go:44] "Start"
Nov 02 01:31:47.299797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.299778    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:47.299875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.299809    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:47.299875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.299825    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:47.302855 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.302832    4912 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:31:47.305491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.305479    4912 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:31:47.305552 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.305502    4912 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:31:47.305552 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.305511    4912 server.go:87] "Starting device plugin registration server"
Nov 02 01:31:47.305808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.305797    4912 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:31:47.305843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.305811    4912 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:31:47.305947 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.305925    4912 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:31:47.306087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.306035    4912 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:31:47.306170 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.306097    4912 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:31:47.306170 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.306104    4912 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:31:47.306270 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.306135    4912 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:31:47.306086634Z","UUID":"77858514-9829-484b-9bc0-c3f4d0c4bde4","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:31:47.307478 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.307455    4912 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:31:47.307623 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.307482    4912 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:31:47.307623 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.307551    4912 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:31:47.307623 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.307584    4912 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-33.us-west-1.compute.internal\" not found"
Nov 02 01:31:47.317270 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.317236    4912 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"]
Nov 02 01:31:47.317362 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.317355    4912 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:47.318211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.318195    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:47.318290 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.318222    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:47.318290 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.318232    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:47.318509 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.318498    4912 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:47.318632 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.318620    4912 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:47.319243 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319225    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:47.319329 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319256    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:47.319329 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319272    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:47.319441 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319228    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:47.319441 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319374    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:47.319441 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319385    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:47.319576 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.319522    4912 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-33.us-west-1.compute.internal\" not found" node="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.319710 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319682    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"95fc15b85861deeb88c7e9a1573ac9e6","Type":"ContainerStarted","Data":"a8fc6fd72aa58d7277a892a5f790b2d2c7da8b4f3326f6f66e757042303ee482"}
Nov 02 01:31:47.319781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319718    4912 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:47.319781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319724    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"95fc15b85861deeb88c7e9a1573ac9e6","Type":"ContainerDied","Data":"a90397d2c85a500c6b08b085d31066513f68c9457082b69ca36ed98a88caea6c"}
Nov 02 01:31:47.319781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319746    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"95fc15b85861deeb88c7e9a1573ac9e6","Type":"ContainerStarted","Data":"6cb6b5398a193c429473f822e2aca3b4a96addbdabbf64994373462fff87c691"}
Nov 02 01:31:47.320412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319798    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"de43b58832dec4b7f2ea5d9fc15899cf","Type":"ContainerStarted","Data":"a29aa03df596f6bee6ece859190af9a79b572c06c69b87adbe381bddeef7911b"}
Nov 02 01:31:47.320412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.319818    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal" event={"ID":"de43b58832dec4b7f2ea5d9fc15899cf","Type":"ContainerStarted","Data":"3d775d12f8252fb7e43a677dc50a3dfcc8672c5f19f26fcb464538f80298c5ef"}
Nov 02 01:31:47.320482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.320431    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:47.320482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.320455    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:47.320482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.320464    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:47.320589 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.320578    4912 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-33.us-west-1.compute.internal\" not found" node="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.377537 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.377467    4912 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 01:31:47.377537 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.377486    4912 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:47.377537 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.377512    4912 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:47.377917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.377900    4912 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:47.386115 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.386094    4912 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.406266 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.406247    4912 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:47.407254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.407238    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:47.407357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.407267    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:47.407357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.407277    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:47.407357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.407318    4912 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.427552 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.427536    4912 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:47.468560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.468537    4912 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:31:47.468656 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.468586    4912 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.468656 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.468599    4912 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.477856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.477827    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de43b58832dec4b7f2ea5d9fc15899cf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"de43b58832dec4b7f2ea5d9fc15899cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.477927 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.477858    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.477927 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.477877    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.520714 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.520692    4912 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.520818 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.520757    4912 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.521332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.521315    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeNotReady"
Nov 02 01:31:47.521432 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.521351    4912 setters.go:546] "Node became not ready" node="ip-10-0-135-33.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T01:31:47Z","lastTransitionTime":"2026-11-02T01:31:47Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 02 01:31:47.539651 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.539631    4912 nodeinfomanager.go:471] existing CSINode "ip-10-0-135-33.us-west-1.compute.internal" is owned by different node (oldNodeID="20c68377-cb5e-4031-81b7-7d3fc102f0a3", newNodeID=""), cleaning up...
Nov 02 01:31:47.546613 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.546598    4912 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:31:47.546714 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:47.546637    4912 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.578962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.578943    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.579379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.578969    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.579379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.578986    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de43b58832dec4b7f2ea5d9fc15899cf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"de43b58832dec4b7f2ea5d9fc15899cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.579379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.579028    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de43b58832dec4b7f2ea5d9fc15899cf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"de43b58832dec4b7f2ea5d9fc15899cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.579379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.579045    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:47.579379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:47.579072    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc15b85861deeb88c7e9a1573ac9e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" (UID: \"95fc15b85861deeb88c7e9a1573ac9e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:48.159961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.159865    4912 apiserver.go:52] "Watching apiserver"
Nov 02 01:31:48.269995 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.269970    4912 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:48.270179 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.270139    4912 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:48.314996 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.314970    4912 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:31:48.317826 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.316844    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq","openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn","openshift-image-registry/image-registry-57b8676cdd-mw579","kube-system/global-pull-secret-syncer-nfs6f","openshift-image-registry/node-ca-9nqbc","openshift-console-operator/console-operator-7cd56b8dbc-2m8ck","openshift-ingress/router-default-7948cc5b95-b87qg","openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw","openshift-multus/multus-mfklz","kube-system/kas-connection-checker-685895c7c9-57lbv","openshift-cluster-node-tuning-operator/tuned-6vlvp","openshift-dns/dns-default-sj5db","kube-system/konnectivity-agent-f2jxf","kube-system/kas-connection-checker-685895c7c9-xkcjd","openshift-multus/network-metrics-daemon-85l6c","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs","openshift-multus/multus-additional-cni-plugins-z9hpd","openshift-network-console/networking-console-plugin-86c478bb88-qxgb8","kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal","openshift-insights/insights-runtime-extractor-dtdwx","kube-system/kas-connection-checker-685895c7c9-5tvxk","openshift-image-registry/image-registry-799447c5f7-7ffhs","openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw","openshift-ingress-canary/ingress-canary-5tjm5","openshift-ovn-kubernetes/ovnkube-node-fxgd2","openshift-network-operator/iptables-alerter-kw282","openshift-network-diagnostics/network-check-target-hqwck","openshift-dns/node-resolver-cz4kl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal","openshift-insights/insights-operator-7c4986d449-bv7fm","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"]
Nov 02 01:31:48.320132 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.320111    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.320473 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.320459    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:48.320814 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.320793    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.321997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.321980    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:48.322663 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.322646    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:48.323246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.323232    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:48.329008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.328992    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.330128 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.330111    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:48.330304 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.330287    4912 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:31:48.330710 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.330698    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.330996 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.330978    4912 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:31:48.363300 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.363272    4912 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:48.363300 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.363289    4912 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal"
Nov 02 01:31:48.370430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.370407    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.370430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.370409    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.370573 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.370520    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.370796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.370779    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.371082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.371051    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 01:31:48.372814 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.372800    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 01:31:48.372851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.372806    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 01:31:48.372851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.372823    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 01:31:48.372851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.372813    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 01:31:48.372955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.372866    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 01:31:48.372997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.372976    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 01:31:48.372997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.372982    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 01:31:48.373234 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.373028    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 01:31:48.373753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.373737    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 01:31:48.373884 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.373866    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 01:31:48.373925 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.373883    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 01:31:48.373925 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.373902    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 01:31:48.374005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.373927    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 01:31:48.374050 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.374010    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.374050 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.374042    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 01:31:48.374144 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.374041    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.374144 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.374095    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.374628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.374604    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.374628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.374626    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 01:31:48.374776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.374642    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.374776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.374645    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 01:31:48.375546 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.375521    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.375736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.375722    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 01:31:48.376389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.376370    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 01:31:48.376646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.376630    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:31:48.376842 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.376827    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.377033 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.377020    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.377213 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.377200    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.377451 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.377437    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:31:48.377563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.377550    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.378007 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.377992    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.378209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.378195    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.378305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.378292    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.378335 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.378310    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.378578 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.378561    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.378866 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.378851    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 01:31:48.378901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.378857    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.380177 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.380160    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.386552 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.386536    4912 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:31:48.419617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.419549    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 01:31:48.419617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.419555    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.423837 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.423821    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 01:31:48.425228 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.425195    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 01:31:48.425310 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.425232    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 01:31:48.470540 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.470509    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:31:48.470540 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.470538    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:31:48.470776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.470668    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.470776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.470754    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.470776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.470759    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.470938 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.470826    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:31:48.471150 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471131    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.471241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471165    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.471241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471187    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.471241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471202    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:31:48.471241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471217    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.471241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471220    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471256    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471259    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471261    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471287    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471270    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471311    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471319    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471329    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471333    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471348    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471372    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471415    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:31:48.471479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471452    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:31:48.472076 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471569    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:48.472076 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471861    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:31:48.472076 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.471978    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:31:48.472076 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472039    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-42sf2\""
Nov 02 01:31:48.472281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472104    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6qvz\""
Nov 02 01:31:48.472370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472356    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:31:48.472444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472380    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 01:31:48.472444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472438    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xv9qq\""
Nov 02 01:31:48.472536 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472437    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:31:48.472536 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472495    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 01:31:48.472536 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472514    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 01:31:48.472536 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472528    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lljjm\""
Nov 02 01:31:48.472694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472558    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fxpjk\""
Nov 02 01:31:48.472694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472568    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n6fsn\""
Nov 02 01:31:48.472694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472589    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qn9km\""
Nov 02 01:31:48.472694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472616    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m64v5\""
Nov 02 01:31:48.472694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472627    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vt6x2\""
Nov 02 01:31:48.472694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472686    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wfswr\""
Nov 02 01:31:48.472922 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472744    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6csd8\""
Nov 02 01:31:48.472922 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472834    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:31:48.472922 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472916    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xbzw4\""
Nov 02 01:31:48.473037 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472925    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-846kb\""
Nov 02 01:31:48.473037 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.472974    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-87db6\""
Nov 02 01:31:48.473037 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.473028    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xjcn9\""
Nov 02 01:31:48.473215 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.473139    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kk5lr\""
Nov 02 01:31:48.473273 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.473258    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j52qq\""
Nov 02 01:31:48.483994 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.483973    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42z6t\" (UniqueName: \"kubernetes.io/projected/277a3a65-7e29-4cb7-a913-d364040935b0-kube-api-access-42z6t\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:48.484094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.483998    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-daemon-config\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.484094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484019    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-registration-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.484094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484035    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tstcv\" (UniqueName: \"kubernetes.io/projected/adda0d83-929a-4e90-a939-49ba1f7f73b4-kube-api-access-tstcv\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.484094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484050    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-kubelet\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.484269 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484144    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aba37fc-45ec-4861-b481-f022d9727476-agent-certs\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:48.484269 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484238    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7xqs\" (UniqueName: \"kubernetes.io/projected/444113a1-c287-4a62-98ba-7b88fe123732-kube-api-access-x7xqs\") pod \"kas-connection-checker-685895c7c9-5tvxk\" (UID: \"444113a1-c287-4a62-98ba-7b88fe123732\") " pod="kube-system/kas-connection-checker-685895c7c9-5tvxk"
Nov 02 01:31:48.484425 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484403    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1284f23-412e-4a65-970e-25ec56f3da4a-serving-cert\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.484491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484459    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-daemon-config\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.484491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484460    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.484570 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484503    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5dl\" (UniqueName: \"kubernetes.io/projected/5c414b43-ca48-4f3e-a38a-c82f2cc27283-kube-api-access-wq5dl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:48.484570 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484534    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-config\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:48.484570 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484563    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.484678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484570    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1284f23-412e-4a65-970e-25ec56f3da4a-serving-cert\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.484678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484536    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aba37fc-45ec-4861-b481-f022d9727476-agent-certs\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:48.484678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484626    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjcgz\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.484678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484662    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfl22\" (UniqueName: \"kubernetes.io/projected/77d4f2e6-e448-4f1e-b817-50cb130f99c1-kube-api-access-gfl22\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:48.484821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484743    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-config\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:48.484821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484775    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-modprobe-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.484821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484797    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.484952 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484822    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:48.484952 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484844    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-bin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.484952 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484868    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovn-node-metrics-cert\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.484952 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484900    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt47v\" (UniqueName: \"kubernetes.io/projected/f74b04ce-df4a-46d3-9eb5-52464d2b5e50-kube-api-access-jt47v\") pod \"kas-connection-checker-685895c7c9-57lbv\" (UID: \"f74b04ce-df4a-46d3-9eb5-52464d2b5e50\") " pod="kube-system/kas-connection-checker-685895c7c9-57lbv"
Nov 02 01:31:48.484952 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484931    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-run\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.485154 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.484990    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.485154 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485046    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.485154 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485125    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-config\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.485154 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485124    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovn-node-metrics-cert\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.485360 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485188    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgnpm\" (UniqueName: \"kubernetes.io/projected/48e3609b-a3dc-4890-aab1-e4ecd697f718-kube-api-access-tgnpm\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:48.485360 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485251    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.485360 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485289    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-host\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.485360 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485355    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-config\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.485512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485370    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgxf\" (UniqueName: \"kubernetes.io/projected/6c3d1051-a370-4a67-bebf-78d6cd902101-kube-api-access-lqgxf\") pod \"migrator-66df5d496b-6brtq\" (UID: \"6c3d1051-a370-4a67-bebf-78d6cd902101\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"
Nov 02 01:31:48.485512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485408    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c414b43-ca48-4f3e-a38a-c82f2cc27283-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:48.485512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485432    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-socket-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.485512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485450    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.485512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485507    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c414b43-ca48-4f3e-a38a-c82f2cc27283-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:48.485738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485550    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqp2\" (UniqueName: \"kubernetes.io/projected/5826852c-0e9b-4031-8213-94051d31c42c-kube-api-access-fjqp2\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.485738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485575    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.485738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485592    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-cni-binary-copy\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.485738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485622    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-socket-dir-parent\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.485738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485647    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-config\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:48.485738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485666    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-log-socket\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.485738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485687    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.485738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485731    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzsd\" (UniqueName: \"kubernetes.io/projected/a9e45b19-94de-40fc-81c4-960324b99e20-kube-api-access-tpzsd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485768    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7tt\" (UniqueName: \"kubernetes.io/projected/1b8b7f85-d24b-4ee9-a036-54554ab354ec-kube-api-access-qr7tt\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485794    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a4994-d804-4369-916f-e9d14143f0c1-tmp-dir\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485796    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98ad998e-3ed5-4892-b6d1-f14e516be75a-cni-binary-copy\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485812    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-config\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485820    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-system-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485864    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485947    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7121a072-9bc5-41fb-920e-bff50c7cd95f-crio-socket\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485975    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.485977    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a4994-d804-4369-916f-e9d14143f0c1-tmp-dir\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486015    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p22\" (UniqueName: \"kubernetes.io/projected/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-kube-api-access-v4p22\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:48.486098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486045    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-kubelet\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486120    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-tmp\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486158    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-netd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486195    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-tmp\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486238    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77d4f2e6-e448-4f1e-b817-50cb130f99c1-cert\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486262    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-slash\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486279    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysconfig\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486314    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486344    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-multus-certs\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486365    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-dbus\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486386    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77d4f2e6-e448-4f1e-b817-50cb130f99c1-cert\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486412    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-device-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486428    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-etc-tuned\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486444    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.486511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486490    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/445a4994-d804-4369-916f-e9d14143f0c1-hosts-file\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486560    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-conf-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486574    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-etc-tuned\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486591    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbde9de9-c649-40ae-a43e-896097fc7dcd-iptables-alerter-script\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486623    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486666    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486727    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgxv\" (UniqueName: \"kubernetes.io/projected/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-kube-api-access-7tgxv\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486775    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486822    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbde9de9-c649-40ae-a43e-896097fc7dcd-iptables-alerter-script\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486855    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlm9\" (UniqueName: \"kubernetes.io/projected/944b4b8f-a639-4153-a4f7-3dc0e5691225-kube-api-access-5hlm9\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486894    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486922    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486930    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486953    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7121a072-9bc5-41fb-920e-bff50c7cd95f-data-volume\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.486980    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aba37fc-45ec-4861-b481-f022d9727476-konnectivity-ca\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:48.487013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487005    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-multus\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487021    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7121a072-9bc5-41fb-920e-bff50c7cd95f-data-volume\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487030    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-ovn\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487077    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e3609b-a3dc-4890-aab1-e4ecd697f718-host\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487104    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487131    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487155    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aba37fc-45ec-4861-b481-f022d9727476-konnectivity-ca\") pod \"konnectivity-agent-f2jxf\" (UID: \"2aba37fc-45ec-4861-b481-f022d9727476\") " pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487245    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-tmp-dir\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487265    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-systemd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487281    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-env-overrides\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487304    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-tmp-dir\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487311    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487337    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487340    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8b7f85-d24b-4ee9-a036-54554ab354ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487365    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-cnibin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487382    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-env-overrides\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487394    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487399    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-kubernetes\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487447    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487481    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487498    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487544    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-sys-fs\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487571    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487597    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487638    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487641    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cnibin\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487642    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.487682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487680    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-k8s-cni-cncf-io\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487712    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b9vc\" (UniqueName: \"kubernetes.io/projected/c1284f23-412e-4a65-970e-25ec56f3da4a-kube-api-access-6b9vc\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487740    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487764    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487783    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-config-volume\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487871    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw9cx\" (UniqueName: \"kubernetes.io/projected/4d375cf0-b26a-45d5-b6ab-23f752812dd5-kube-api-access-pw9cx\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487884    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487917    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-systemd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.487987    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-config-volume\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488018    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-default-certificate\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488049    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x422d\" (UniqueName: \"kubernetes.io/projected/445a4994-d804-4369-916f-e9d14143f0c1-kube-api-access-x422d\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488102    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488128    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-script-lib\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488169    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-tmp\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488198    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-stats-auth\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488217    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-default-certificate\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488226    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-snapshots\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488238    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e45b19-94de-40fc-81c4-960324b99e20-tmp\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488252    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-systemd-units\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488277    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c414b43-ca48-4f3e-a38a-c82f2cc27283-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488305    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-system-cni-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488347    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztztb\" (UniqueName: \"kubernetes.io/projected/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-api-access-ztztb\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488380    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-ovnkube-script-lib\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488378    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zwf\" (UniqueName: \"kubernetes.io/projected/98ad998e-3ed5-4892-b6d1-f14e516be75a-kube-api-access-25zwf\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488398    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c1284f23-412e-4a65-970e-25ec56f3da4a-snapshots\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488402    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-stats-auth\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488428    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69bj\" (UniqueName: \"kubernetes.io/projected/fbde9de9-c649-40ae-a43e-896097fc7dcd-kube-api-access-z69bj\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488457    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnvdl\" (UniqueName: \"kubernetes.io/projected/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-kube-api-access-cnvdl\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488485    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/277a3a65-7e29-4cb7-a913-d364040935b0-serving-cert\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488515    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-service-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488542    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-var-lib-kubelet\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488514    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c414b43-ca48-4f3e-a38a-c82f2cc27283-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488567    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488610    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ppz\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488618    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/277a3a65-7e29-4cb7-a913-d364040935b0-serving-cert\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488659    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-service-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488707    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488797    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kn5\" (UniqueName: \"kubernetes.io/projected/230459c8-baa4-47cc-9cc2-73f0535fb650-kube-api-access-r8kn5\") pod \"kas-connection-checker-685895c7c9-xkcjd\" (UID: \"230459c8-baa4-47cc-9cc2-73f0535fb650\") " pod="kube-system/kas-connection-checker-685895c7c9-xkcjd"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488821    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488855    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488875    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-netns\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488889    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-sys\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488904    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:48.488892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488924    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-etc-kubernetes\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488948    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-serving-cert\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.488969    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-etc-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489008    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-bin\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489042    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58kn4\" (UniqueName: \"kubernetes.io/projected/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-kube-api-access-58kn4\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489081    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489109    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-serving-cert\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489145    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-kubelet-config\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489164    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489245    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489288    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489354    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/277a3a65-7e29-4cb7-a913-d364040935b0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489397    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489404    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-conf\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489442    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-lib-modules\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489470    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1284f23-412e-4a65-970e-25ec56f3da4a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489471    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489478    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-var-lib-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489514    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489533    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489569    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489593    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-netns\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489636    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-hostroot\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489654    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489670    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbde9de9-c649-40ae-a43e-896097fc7dcd-host-slash\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489746    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsgc\" (UniqueName: \"kubernetes.io/projected/1b3aaf5a-58ac-4fa6-846e-0764a4803577-kube-api-access-ncsgc\") pod \"network-check-source-6f78d59f99-mvlvw\" (UID: \"1b3aaf5a-58ac-4fa6-846e-0764a4803577\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489786    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxgq\" (UniqueName: \"kubernetes.io/projected/f3c0e9d9-eda0-4e8f-8677-0bd4406243a4-kube-api-access-4mxgq\") pod \"volume-data-source-validator-58c9f5f46d-888h6\" (UID: \"f3c0e9d9-eda0-4e8f-8677-0bd4406243a4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489821    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447b6716-29a3-4000-8df4-022f92de0e23-original-pull-secret\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489820    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26929d97-925d-4839-9bd7-e0fc4b3373a0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489853    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489870    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-node-log\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489887    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48e3609b-a3dc-4890-aab1-e4ecd697f718-serviceca\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489912    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.489966    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26929d97-925d-4839-9bd7-e0fc4b3373a0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.490119    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48e3609b-a3dc-4890-aab1-e4ecd697f718-serviceca\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:48.490493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.490235    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.545012 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.544773    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5dl\" (UniqueName: \"kubernetes.io/projected/5c414b43-ca48-4f3e-a38a-c82f2cc27283-kube-api-access-wq5dl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4zktv\" (UID: \"5c414b43-ca48-4f3e-a38a-c82f2cc27283\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv"
Nov 02 01:31:48.548586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.545174    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgnpm\" (UniqueName: \"kubernetes.io/projected/48e3609b-a3dc-4890-aab1-e4ecd697f718-kube-api-access-tgnpm\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:48.548586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.545274    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgxf\" (UniqueName: \"kubernetes.io/projected/6c3d1051-a370-4a67-bebf-78d6cd902101-kube-api-access-lqgxf\") pod \"migrator-66df5d496b-6brtq\" (UID: \"6c3d1051-a370-4a67-bebf-78d6cd902101\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq"
Nov 02 01:31:48.548586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.545326    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjcgz\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.548586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.545342    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7xqs\" (UniqueName: \"kubernetes.io/projected/444113a1-c287-4a62-98ba-7b88fe123732-kube-api-access-x7xqs\") pod \"kas-connection-checker-685895c7c9-5tvxk\" (UID: \"444113a1-c287-4a62-98ba-7b88fe123732\") " pod="kube-system/kas-connection-checker-685895c7c9-5tvxk"
Nov 02 01:31:48.548586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.545547    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt47v\" (UniqueName: \"kubernetes.io/projected/f74b04ce-df4a-46d3-9eb5-52464d2b5e50-kube-api-access-jt47v\") pod \"kas-connection-checker-685895c7c9-57lbv\" (UID: \"f74b04ce-df4a-46d3-9eb5-52464d2b5e50\") " pod="kube-system/kas-connection-checker-685895c7c9-57lbv"
Nov 02 01:31:48.548586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.545685    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tstcv\" (UniqueName: \"kubernetes.io/projected/adda0d83-929a-4e90-a939-49ba1f7f73b4-kube-api-access-tstcv\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.548586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.545719    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7tt\" (UniqueName: \"kubernetes.io/projected/1b8b7f85-d24b-4ee9-a036-54554ab354ec-kube-api-access-qr7tt\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.548586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.545737    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42z6t\" (UniqueName: \"kubernetes.io/projected/277a3a65-7e29-4cb7-a913-d364040935b0-kube-api-access-42z6t\") pod \"console-operator-7cd56b8dbc-2m8ck\" (UID: \"277a3a65-7e29-4cb7-a913-d364040935b0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:48.548586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.545814    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzsd\" (UniqueName: \"kubernetes.io/projected/a9e45b19-94de-40fc-81c4-960324b99e20-kube-api-access-tpzsd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.549559 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.549535    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p22\" (UniqueName: \"kubernetes.io/projected/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-kube-api-access-v4p22\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:48.549559 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.549550    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.549793 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.549771    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgxv\" (UniqueName: \"kubernetes.io/projected/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-kube-api-access-7tgxv\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:48.549840 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.549771    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqp2\" (UniqueName: \"kubernetes.io/projected/5826852c-0e9b-4031-8213-94051d31c42c-kube-api-access-fjqp2\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.549872 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.549851    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlm9\" (UniqueName: \"kubernetes.io/projected/944b4b8f-a639-4153-a4f7-3dc0e5691225-kube-api-access-5hlm9\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:48.550200 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.550166    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw9cx\" (UniqueName: \"kubernetes.io/projected/4d375cf0-b26a-45d5-b6ab-23f752812dd5-kube-api-access-pw9cx\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:48.550511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.550472    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b9vc\" (UniqueName: \"kubernetes.io/projected/c1284f23-412e-4a65-970e-25ec56f3da4a-kube-api-access-6b9vc\") pod \"insights-operator-7c4986d449-bv7fm\" (UID: \"c1284f23-412e-4a65-970e-25ec56f3da4a\") " pod="openshift-insights/insights-operator-7c4986d449-bv7fm"
Nov 02 01:31:48.550511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.550491    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnvdl\" (UniqueName: \"kubernetes.io/projected/8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4-kube-api-access-cnvdl\") pod \"service-ca-operator-544d749c8c-xm8rw\" (UID: \"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw"
Nov 02 01:31:48.550604 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.550550    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zwf\" (UniqueName: \"kubernetes.io/projected/98ad998e-3ed5-4892-b6d1-f14e516be75a-kube-api-access-25zwf\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.550604 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.550554    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69bj\" (UniqueName: \"kubernetes.io/projected/fbde9de9-c649-40ae-a43e-896097fc7dcd-kube-api-access-z69bj\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:48.550834 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.550817    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfl22\" (UniqueName: \"kubernetes.io/projected/77d4f2e6-e448-4f1e-b817-50cb130f99c1-kube-api-access-gfl22\") pod \"ingress-canary-5tjm5\" (UID: \"77d4f2e6-e448-4f1e-b817-50cb130f99c1\") " pod="openshift-ingress-canary/ingress-canary-5tjm5"
Nov 02 01:31:48.551332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.551317    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ppz\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.551383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.551369    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lx6\" (UniqueName: \"kubernetes.io/projected/08bd96c5-7bd3-4b79-847c-fd7f7866e231-kube-api-access-28lx6\") pod \"network-check-target-hqwck\" (UID: \"08bd96c5-7bd3-4b79-847c-fd7f7866e231\") " pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:48.553351 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.553333    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztztb\" (UniqueName: \"kubernetes.io/projected/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-api-access-ztztb\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.553719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.553703    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x422d\" (UniqueName: \"kubernetes.io/projected/445a4994-d804-4369-916f-e9d14143f0c1-kube-api-access-x422d\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:48.554255 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.554240    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxgq\" (UniqueName: \"kubernetes.io/projected/f3c0e9d9-eda0-4e8f-8677-0bd4406243a4-kube-api-access-4mxgq\") pod \"volume-data-source-validator-58c9f5f46d-888h6\" (UID: \"f3c0e9d9-eda0-4e8f-8677-0bd4406243a4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-888h6"
Nov 02 01:31:48.555000 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.554963    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.555319 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.555301    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58kn4\" (UniqueName: \"kubernetes.io/projected/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-kube-api-access-58kn4\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.555588 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.555569    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsgc\" (UniqueName: \"kubernetes.io/projected/1b3aaf5a-58ac-4fa6-846e-0764a4803577-kube-api-access-ncsgc\") pod \"network-check-source-6f78d59f99-mvlvw\" (UID: \"1b3aaf5a-58ac-4fa6-846e-0764a4803577\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvlvw"
Nov 02 01:31:48.556313 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.556296    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kn5\" (UniqueName: \"kubernetes.io/projected/230459c8-baa4-47cc-9cc2-73f0535fb650-kube-api-access-r8kn5\") pod \"kas-connection-checker-685895c7c9-xkcjd\" (UID: \"230459c8-baa4-47cc-9cc2-73f0535fb650\") " pod="kube-system/kas-connection-checker-685895c7c9-xkcjd"
Nov 02 01:31:48.590925 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.590898    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-slash\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.590925 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.590926    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysconfig\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.590941    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.590957    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-multus-certs\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.590971    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-dbus\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.590987    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-device-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591002    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/445a4994-d804-4369-916f-e9d14143f0c1-hosts-file\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591025    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysconfig\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591079    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-multus-certs\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591086    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/445a4994-d804-4369-916f-e9d14143f0c1-hosts-file\") pod \"node-resolver-cz4kl\" (UID: \"445a4994-d804-4369-916f-e9d14143f0c1\") " pod="openshift-dns/node-resolver-cz4kl"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591088    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-dbus\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591078    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591083    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-conf-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591025    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-slash\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591024    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-conf-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591116    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-device-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591152    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591176    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.591243    4912 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.591285    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.091270575 +0000 UTC m=+1.996735922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591303    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591324    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-multus\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591340    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-ovn\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.591349    4912 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591356    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e3609b-a3dc-4890-aab1-e4ecd697f718-host\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.591377    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.091365541 +0000 UTC m=+1.996830875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591391    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e3609b-a3dc-4890-aab1-e4ecd697f718-host\") pod \"node-ca-9nqbc\" (UID: \"48e3609b-a3dc-4890-aab1-e4ecd697f718\") " pod="openshift-image-registry/node-ca-9nqbc"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591393    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-systemd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591415    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-multus\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.591437    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.091411577 +0000 UTC m=+1.996876912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591444    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-ovn\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591453    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-cnibin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591488    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-kubernetes\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591518    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-sys-fs\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591523    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-kubernetes\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591545    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591493    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-cnibin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591562    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-sys-fs\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591490    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-systemd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591573    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cnibin\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591586    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591598    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-k8s-cni-cncf-io\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.591612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591624    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-systemd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591645    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-cnibin\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591657    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591678    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-systemd-units\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591693    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-system-cni-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591719    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-var-lib-kubelet\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591743    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591747    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-k8s-cni-cncf-io\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591750    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-systemd-units\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591766    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-system-cni-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591767    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591792    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-systemd\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591796    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591798    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-var-lib-kubelet\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.591771    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.591830    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591832    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-netns\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591802    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-run-netns\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591889    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.591901    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.091888937 +0000 UTC m=+1.997354291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591914    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-sys\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591932    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-etc-kubernetes\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591960    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-etc-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591965    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-sys\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591969    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-etc-kubernetes\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.591984    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-bin\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592004    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-kubelet-config\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592019    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-conf\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592032    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-lib-modules\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592051    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-var-lib-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592072    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-etc-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592091    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447b6716-29a3-4000-8df4-022f92de0e23-kubelet-config\") pod \"global-pull-secret-syncer-nfs6f\" (UID: \"447b6716-29a3-4000-8df4-022f92de0e23\") " pod="kube-system/global-pull-secret-syncer-nfs6f"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592107    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-bin\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592115    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-var-lib-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592129    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592148    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-lib-modules\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592148    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592178    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-netns\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592196    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-sysctl-conf\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592208    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-hostroot\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592211    4912 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592234    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbde9de9-c649-40ae-a43e-896097fc7dcd-host-slash\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592241    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-hostroot\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592246    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-run-netns\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592244    4912 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592257    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.092247918 +0000 UTC m=+1.997713254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : secret "metrics-daemon-secret" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592285    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbde9de9-c649-40ae-a43e-896097fc7dcd-host-slash\") pod \"iptables-alerter-kw282\" (UID: \"fbde9de9-c649-40ae-a43e-896097fc7dcd\") " pod="openshift-network-operator/iptables-alerter-kw282"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592296    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592315    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-node-log\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592340    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.092330724 +0000 UTC m=+1.997796058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592341    4912 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592351    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-node-log\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592356    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-registration-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592375    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-kubelet\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592385    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.092375384 +0000 UTC m=+1.997840718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592398    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-kubelet\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592406    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592424    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592423    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-registration-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592441    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-modprobe-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592456    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592460    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592471    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592466    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-run-openvswitch\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592499    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-bin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592532    4912 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592538    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-cni-bin\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592551    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-etc-modprobe-d\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592573    4912 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592581    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-run\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592534    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-run\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592584    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.092571815 +0000 UTC m=+1.998037170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : configmap "kube-rbac-proxy" not found
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592616    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.592563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592628    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.092614886 +0000 UTC m=+1.998080222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592645    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592657    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-host\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592684    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-socket-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592707    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e45b19-94de-40fc-81c4-960324b99e20-host\") pod \"tuned-6vlvp\" (UID: \"a9e45b19-94de-40fc-81c4-960324b99e20\") " pod="openshift-cluster-node-tuning-operator/tuned-6vlvp"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592709    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592738    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592745    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/adda0d83-929a-4e90-a939-49ba1f7f73b4-socket-dir\") pod \"aws-ebs-csi-driver-node-x9fcw\" (UID: \"adda0d83-929a-4e90-a939-49ba1f7f73b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9fcw"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592762    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-socket-dir-parent\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592802    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8b7f85-d24b-4ee9-a036-54554ab354ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9hpd\" (UID: \"1b8b7f85-d24b-4ee9-a036-54554ab354ec\") " pod="openshift-multus/multus-additional-cni-plugins-z9hpd"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592811    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-multus-socket-dir-parent\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592793    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592832    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592836    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-log-socket\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592855    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-system-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592869    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7121a072-9bc5-41fb-920e-bff50c7cd95f-crio-socket\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592883    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.092871231 +0000 UTC m=+1.998336570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592906    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7121a072-9bc5-41fb-920e-bff50c7cd95f-crio-socket\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592911    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592917    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-log-socket\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592939    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-kubelet\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592948    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-system-cni-dir\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592968    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-netd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.592986    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ad998e-3ed5-4892-b6d1-f14e516be75a-host-var-lib-kubelet\") pod \"multus-mfklz\" (UID: \"98ad998e-3ed5-4892-b6d1-f14e516be75a\") " pod="openshift-multus/multus-mfklz"
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.592991    4912 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:48.593038    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:49.093028177 +0000 UTC m=+1.998493512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:48.594769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.593037    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce-host-cni-netd\") pod \"ovnkube-node-fxgd2\" (UID: \"5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:48.620323 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.620299    4912 scope.go:122] "RemoveContainer" containerID="0b2f329e986fdd8bdd7308eed4623b842d3585f482c0e1cf35ea45cc222a453f"
Nov 02 01:31:48.632641 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.632627    4912 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:31:48.857590 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:48.857561    4912 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:49.058376 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.058302    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:49.097507 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097474    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:49.097655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097524    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:49.097655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097553    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:49.097655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097578    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:49.097655 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097619    4912 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097658    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097684    4912 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097687    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.097668345 +0000 UTC m=+3.003133697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097734    4912 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097737    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097751    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097754    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.097741152 +0000 UTC m=+3.003206490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097775    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.097764852 +0000 UTC m=+3.003230187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097798    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.097784852 +0000 UTC m=+3.003250187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:49.097850 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097814    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.097809087 +0000 UTC m=+3.003274421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097864    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097885    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097908    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097935    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097960    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.097992    4912 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.097996    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098014    4912 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098039    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.098020628 +0000 UTC m=+3.003485980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098049    4912 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098079    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.098047386 +0000 UTC m=+3.003512737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : secret "metrics-daemon-secret" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098113    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.098101489 +0000 UTC m=+3.003566851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098118    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098122    4912 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098130    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098159    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.098152227 +0000 UTC m=+3.003617563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:49.098281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.098178    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:50.098167308 +0000 UTC m=+3.003632656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:49.098746 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.098391    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7121a072-9bc5-41fb-920e-bff50c7cd95f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:49.201692 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.201665    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:31:49.275398 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.275372    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/2.log"
Nov 02 01:31:49.275780 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.275762    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/1.log"
Nov 02 01:31:49.275873 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.275801    4912 generic.go:356] "Generic (PLEG): container finished" podID="277a3a65-7e29-4cb7-a913-d364040935b0" containerID="2b5251344cfdde4af8cb9dcd33e96ae60841bd14077772da8a8953244f59cc9e" exitCode=255
Nov 02 01:31:49.275948 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.275900    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" event={"ID":"277a3a65-7e29-4cb7-a913-d364040935b0","Type":"ContainerDied","Data":"2b5251344cfdde4af8cb9dcd33e96ae60841bd14077772da8a8953244f59cc9e"}
Nov 02 01:31:49.275985 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.275951    4912 scope.go:122] "RemoveContainer" containerID="0b2f329e986fdd8bdd7308eed4623b842d3585f482c0e1cf35ea45cc222a453f"
Nov 02 01:31:49.276243 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.276223    4912 scope.go:122] "RemoveContainer" containerID="2b5251344cfdde4af8cb9dcd33e96ae60841bd14077772da8a8953244f59cc9e"
Nov 02 01:31:49.276516 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:49.276480    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m8ck_openshift-console-operator(277a3a65-7e29-4cb7-a913-d364040935b0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0"
Nov 02 01:31:49.288823 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:49.288799    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:50.107383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107346    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:50.107603 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107397    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:50.107603 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107438    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:50.107603 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107477    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:50.107603 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107511    4912 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:31:50.107603 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107585    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.107565087 +0000 UTC m=+5.013030443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : secret "metrics-daemon-secret" not found
Nov 02 01:31:50.107603 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107587    4912 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107642    4912 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107642    4912 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107652    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.107635236 +0000 UTC m=+5.013100583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107512    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107591    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107681    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.10767016 +0000 UTC m=+5.013135498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107682    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107706    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.107700334 +0000 UTC m=+5.013165669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107724    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.107715866 +0000 UTC m=+5.013181204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107770    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107806    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107832    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107850    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:50.107882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.107882    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107958    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.107970    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.108001    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.107989722 +0000 UTC m=+5.013455067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.108052    4912 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.108101    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.108093191 +0000 UTC m=+5.013558527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.108135    4912 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.108163    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.108153484 +0000 UTC m=+5.013618827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.108220    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.108213818 +0000 UTC m=+5.013679152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.108262    4912 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:50.108426 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.108292    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:52.108282918 +0000 UTC m=+5.013748270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:50.281390 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.281355    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6brtq" event={"ID":"6c3d1051-a370-4a67-bebf-78d6cd902101","Type":"ContainerStarted","Data":"aaf21bef672649deb43f59bc14a705189efa66827d69fd988af2e0bb40f7b911"}
Nov 02 01:31:50.283043 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.283022    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/2.log"
Nov 02 01:31:50.283369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.283352    4912 scope.go:122] "RemoveContainer" containerID="2b5251344cfdde4af8cb9dcd33e96ae60841bd14077772da8a8953244f59cc9e"
Nov 02 01:31:50.283563 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:50.283545    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m8ck_openshift-console-operator(277a3a65-7e29-4cb7-a913-d364040935b0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0"
Nov 02 01:31:50.340214 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:50.340184    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:52.125322 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125275    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125344    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125371    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125404    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125441    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125456    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125474    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125482    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125495    4912 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125497    4912 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125511    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125550    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125530229 +0000 UTC m=+9.030995584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125571    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs podName:4d375cf0-b26a-45d5-b6ab-23f752812dd5 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125564626 +0000 UTC m=+9.031029960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs") pod "network-metrics-daemon-85l6c" (UID: "4d375cf0-b26a-45d5-b6ab-23f752812dd5") : secret "metrics-daemon-secret" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125579    4912 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125589    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125580158 +0000 UTC m=+9.031045498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125610    4912 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125626    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125638    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125620983 +0000 UTC m=+9.031086333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125636    4912 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125671    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125658291 +0000 UTC m=+9.031123626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125672    4912 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125614    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125695    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125694    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125699    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls podName:7121a072-9bc5-41fb-920e-bff50c7cd95f nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125682965 +0000 UTC m=+9.031148318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtdwx" (UID: "7121a072-9bc5-41fb-920e-bff50c7cd95f") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125743    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125735243 +0000 UTC m=+9.031200578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125753    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125748003 +0000 UTC m=+9.031213338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:52.125769    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125797    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125785671 +0000 UTC m=+9.031251012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125812    4912 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:52.125847 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:52.125840    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:31:56.125833935 +0000 UTC m=+9.031299270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:53.914115 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:53.914084    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cz4kl_445a4994-d804-4369-916f-e9d14143f0c1/dns-node-resolver/0.log"
Nov 02 01:31:54.550165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:54.550118    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:54.550520 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:54.550505    4912 scope.go:122] "RemoveContainer" containerID="2b5251344cfdde4af8cb9dcd33e96ae60841bd14077772da8a8953244f59cc9e"
Nov 02 01:31:54.550685 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:54.550667    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m8ck_openshift-console-operator(277a3a65-7e29-4cb7-a913-d364040935b0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0"
Nov 02 01:31:54.714326 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:54.714298    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9nqbc_48e3609b-a3dc-4890-aab1-e4ecd697f718/node-ca/0.log"
Nov 02 01:31:54.950362 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:54.950338    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f2jxf"
Nov 02 01:31:55.314158 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:55.314082    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5tjm5_77d4f2e6-e448-4f1e-b817-50cb130f99c1/serve-healthcheck-canary/0.log"
Nov 02 01:31:55.914886 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:55.914854    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6brtq_6c3d1051-a370-4a67-bebf-78d6cd902101/migrator/0.log"
Nov 02 01:31:56.115354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.115322    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-4zktv_5c414b43-ca48-4f3e-a38a-c82f2cc27283/kube-storage-version-migrator-operator/0.log"
Nov 02 01:31:56.165386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165303    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:56.165386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165341    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:31:56.165386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165369    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165411    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165445    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165479    4912 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165549    4912 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165570    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls podName:944b4b8f-a639-4153-a4f7-3dc0e5691225 nodeName:}" failed. No retries permitted until 2026-11-02 01:32:04.165551453 +0000 UTC m=+17.071016802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gbhrs" (UID: "944b4b8f-a639-4153-a4f7-3dc0e5691225") : secret "samples-operator-tls" not found
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165604    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165481    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165625    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799447c5f7-7ffhs: secret "image-registry-tls" not found
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165611    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls podName:4161f42e-a6d1-48ea-b5dc-4d21e16facf6 nodeName:}" failed. No retries permitted until 2026-11-02 01:32:04.165593018 +0000 UTC m=+17.071058354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls") pod "dns-default-sj5db" (UID: "4161f42e-a6d1-48ea-b5dc-4d21e16facf6") : secret "dns-default-metrics-tls" not found
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165549    4912 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:31:56.165672 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165671    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls podName:9404b1b3-624d-4246-a20a-8627cfd58e2c nodeName:}" failed. No retries permitted until 2026-11-02 01:32:04.165657398 +0000 UTC m=+17.071122749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls") pod "image-registry-799447c5f7-7ffhs" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c") : secret "image-registry-tls" not found
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165702    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert podName:26929d97-925d-4839-9bd7-e0fc4b3373a0 nodeName:}" failed. No retries permitted until 2026-11-02 01:32:04.16568654 +0000 UTC m=+17.071151895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qxgb8" (UID: "26929d97-925d-4839-9bd7-e0fc4b3373a0") : secret "networking-console-plugin-cert" not found
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165698    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165737    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165761    4912 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165764    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165797    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls podName:ce9b8562-0baf-437e-bf85-5cc8f0e32de1 nodeName:}" failed. No retries permitted until 2026-11-02 01:32:04.165786761 +0000 UTC m=+17.071252109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gbxrn" (UID: "ce9b8562-0baf-437e-bf85-5cc8f0e32de1") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165779    4912 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165836    4912 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.165832    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165863    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:32:04.165847259 +0000 UTC m=+17.071312610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : configmap references non-existent config key: service-ca.crt
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165887    4912 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165897    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs podName:5826852c-0e9b-4031-8213-94051d31c42c nodeName:}" failed. No retries permitted until 2026-11-02 01:32:04.165886769 +0000 UTC m=+17.071352105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs") pod "router-default-7948cc5b95-b87qg" (UID: "5826852c-0e9b-4031-8213-94051d31c42c") : secret "router-metrics-certs-default" not found
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165898    4912 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b8676cdd-mw579: secret "image-registry-tls" not found
Nov 02 01:31:56.166086 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:56.165933    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls podName:6c29bb7a-6e1f-4b1a-b89f-f73cabca6774 nodeName:}" failed. No retries permitted until 2026-11-02 01:32:04.165922928 +0000 UTC m=+17.071388306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls") pod "image-registry-57b8676cdd-mw579" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774") : secret "image-registry-tls" not found
Nov 02 01:31:56.168039 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.168020    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7121a072-9bc5-41fb-920e-bff50c7cd95f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtdwx\" (UID: \"7121a072-9bc5-41fb-920e-bff50c7cd95f\") " pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:56.168146 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.168042    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d375cf0-b26a-45d5-b6ab-23f752812dd5-metrics-certs\") pod \"network-metrics-daemon-85l6c\" (UID: \"4d375cf0-b26a-45d5-b6ab-23f752812dd5\") " pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:56.282215 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.282188    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:56.423627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.423532    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85l6c"
Nov 02 01:31:56.431611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.431584    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtdwx"
Nov 02 01:31:56.792242 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.792162    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-85l6c"]
Nov 02 01:31:56.795114 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:56.795047    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d375cf0_b26a_45d5_b6ab_23f752812dd5.slice/crio-212e7aa62144ca93e826203f34a3255b7a5e0ef88d079cef9002e50c3a209362 WatchSource:0}: Error finding container 212e7aa62144ca93e826203f34a3255b7a5e0ef88d079cef9002e50c3a209362: Status 404 returned error can't find the container with id 212e7aa62144ca93e826203f34a3255b7a5e0ef88d079cef9002e50c3a209362
Nov 02 01:31:56.843462 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:56.843433    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtdwx"]
Nov 02 01:31:56.847555 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:31:56.847514    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7121a072_9bc5_41fb_920e_bff50c7cd95f.slice/crio-3ea1596521df8abe6ec12e302118eb954627a0f44fe1f77006b58acd73e69ba3 WatchSource:0}: Error finding container 3ea1596521df8abe6ec12e302118eb954627a0f44fe1f77006b58acd73e69ba3: Status 404 returned error can't find the container with id 3ea1596521df8abe6ec12e302118eb954627a0f44fe1f77006b58acd73e69ba3
Nov 02 01:31:57.308983 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:57.308942    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtdwx" event={"ID":"7121a072-9bc5-41fb-920e-bff50c7cd95f","Type":"ContainerStarted","Data":"26d9439b07d932a79efb25608ce2bb31a99c14da31467aedef60ee6fa9fe9357"}
Nov 02 01:31:57.309526 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:57.309013    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtdwx" event={"ID":"7121a072-9bc5-41fb-920e-bff50c7cd95f","Type":"ContainerStarted","Data":"3ea1596521df8abe6ec12e302118eb954627a0f44fe1f77006b58acd73e69ba3"}
Nov 02 01:31:57.310174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:57.310134    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85l6c" event={"ID":"4d375cf0-b26a-45d5-b6ab-23f752812dd5","Type":"ContainerStarted","Data":"212e7aa62144ca93e826203f34a3255b7a5e0ef88d079cef9002e50c3a209362"}
Nov 02 01:31:57.921624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:57.921589    4912 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-33.us-west-1.compute.internal" event="NodeReady"
Nov 02 01:31:58.222022 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:58.221944    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:58.279672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:58.279557    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:31:58.279864 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:58.279846    4912 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:31:58.315403 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:58.315377    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtdwx" event={"ID":"7121a072-9bc5-41fb-920e-bff50c7cd95f","Type":"ContainerStarted","Data":"532fcd68a118a4898688ef0d65efd8b49f30e59637da1ad4e6409e4630c39b8b"}
Nov 02 01:31:58.450689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:58.450652    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:58.832268 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:58.832232    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:58.858640 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:58.858612    4912 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:31:58.859089 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:58.859052    4912 scope.go:122] "RemoveContainer" containerID="2b5251344cfdde4af8cb9dcd33e96ae60841bd14077772da8a8953244f59cc9e"
Nov 02 01:31:59.035798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.035772    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:59.237618 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.237584    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:59.325317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.325277    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85l6c" event={"ID":"4d375cf0-b26a-45d5-b6ab-23f752812dd5","Type":"ContainerStarted","Data":"e365f85c1394726299bb39bcf9f7a154b375978822fd72501822216d3ccdefc4"}
Nov 02 01:31:59.325752 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.325377    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85l6c" event={"ID":"4d375cf0-b26a-45d5-b6ab-23f752812dd5","Type":"ContainerStarted","Data":"506f5a37b5a81544f67e8ec252a3f518ab87cb66746b82dc2dc1e2eca448dfc8"}
Nov 02 01:31:59.327078 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.327040    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:31:59.327455 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.327437    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/2.log"
Nov 02 01:31:59.327533 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.327475    4912 generic.go:356] "Generic (PLEG): container finished" podID="277a3a65-7e29-4cb7-a913-d364040935b0" containerID="8877f603d5915124dcf14b371a01e0cf58bdd0d5e059b2fee2693152ca410355" exitCode=255
Nov 02 01:31:59.327533 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.327524    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" event={"ID":"277a3a65-7e29-4cb7-a913-d364040935b0","Type":"ContainerDied","Data":"8877f603d5915124dcf14b371a01e0cf58bdd0d5e059b2fee2693152ca410355"}
Nov 02 01:31:59.327640 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.327569    4912 scope.go:122] "RemoveContainer" containerID="2b5251344cfdde4af8cb9dcd33e96ae60841bd14077772da8a8953244f59cc9e"
Nov 02 01:31:59.327878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.327856    4912 scope.go:122] "RemoveContainer" containerID="8877f603d5915124dcf14b371a01e0cf58bdd0d5e059b2fee2693152ca410355"
Nov 02 01:31:59.328202 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:31:59.328180    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m8ck_openshift-console-operator(277a3a65-7e29-4cb7-a913-d364040935b0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0"
Nov 02 01:31:59.434451 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.434418    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:59.635658 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.635624    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:59.812647 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.812602    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:59.920042 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:31:59.919965    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxgd2"
Nov 02 01:32:00.333162 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:00.333095    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:32:00.836377 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:00.836342    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:01.044940 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:01.044834    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:01.220110 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:01.220080    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:01.338301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:01.338210    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtdwx" event={"ID":"7121a072-9bc5-41fb-920e-bff50c7cd95f","Type":"ContainerStarted","Data":"0182d64cdfa4945d59262658b5292a7b6d42e43ad9c86a403b8529aa63420f45"}
Nov 02 01:32:01.399639 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:01.399609    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:01.577125 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:01.577094    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:01.765650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:01.765620    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:02.607162 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:02.607114    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:02.830160 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:02.830122    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:03.028430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:03.028402    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:03.228252 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:03.228219    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:03.439387 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:03.439357    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:03.634866 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:03.634826    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:04.245192 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.245160    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:32:04.245367 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.245216    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:32:04.245367 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.245242    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:32:04.245367 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.245268    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:32:04.245367 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.245292    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:32:04.245753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.245440    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:32:04.245753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.245488    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:32:04.245753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.245520    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:32:04.246627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.246599    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5826852c-0e9b-4031-8213-94051d31c42c-service-ca-bundle\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:32:04.248272 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.248246    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"image-registry-799447c5f7-7ffhs\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") " pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:32:04.248455 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.248410    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4161f42e-a6d1-48ea-b5dc-4d21e16facf6-metrics-tls\") pod \"dns-default-sj5db\" (UID: \"4161f42e-a6d1-48ea-b5dc-4d21e16facf6\") " pod="openshift-dns/dns-default-sj5db"
Nov 02 01:32:04.248567 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.248549    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26929d97-925d-4839-9bd7-e0fc4b3373a0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qxgb8\" (UID: \"26929d97-925d-4839-9bd7-e0fc4b3373a0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:32:04.248681 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.248662    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/944b4b8f-a639-4153-a4f7-3dc0e5691225-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gbhrs\" (UID: \"944b4b8f-a639-4153-a4f7-3dc0e5691225\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:32:04.248863 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.248844    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"image-registry-57b8676cdd-mw579\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") " pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:32:04.248898 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.248857    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce9b8562-0baf-437e-bf85-5cc8f0e32de1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gbxrn\" (UID: \"ce9b8562-0baf-437e-bf85-5cc8f0e32de1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:32:04.248898 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.248844    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5826852c-0e9b-4031-8213-94051d31c42c-metrics-certs\") pod \"router-default-7948cc5b95-b87qg\" (UID: \"5826852c-0e9b-4031-8213-94051d31c42c\") " pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:32:04.521094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.520983    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"
Nov 02 01:32:04.521262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.521114    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:32:04.521262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.521160    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:32:04.523122 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.523104    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"
Nov 02 01:32:04.523122 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.523115    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"
Nov 02 01:32:04.529386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.529309    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:32:04.530380 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.530367    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sj5db"
Nov 02 01:32:04.551107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.551081    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:32:04.551545 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.551518    4912 scope.go:122] "RemoveContainer" containerID="8877f603d5915124dcf14b371a01e0cf58bdd0d5e059b2fee2693152ca410355"
Nov 02 01:32:04.551756 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:32:04.551713    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m8ck_openshift-console-operator(277a3a65-7e29-4cb7-a913-d364040935b0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0"
Nov 02 01:32:04.720800 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.720766    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b8676cdd-mw579"]
Nov 02 01:32:04.795806 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.795725    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:04.929193 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.929171    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qxgb8"]
Nov 02 01:32:04.931726 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:32:04.931697    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26929d97_925d_4839_9bd7_e0fc4b3373a0.slice/crio-0c68f0723fa9575a61e22a310c9d8c254a48a1e3ca12a0546e9976996751f85c WatchSource:0}: Error finding container 0c68f0723fa9575a61e22a310c9d8c254a48a1e3ca12a0546e9976996751f85c: Status 404 returned error can't find the container with id 0c68f0723fa9575a61e22a310c9d8c254a48a1e3ca12a0546e9976996751f85c
Nov 02 01:32:04.962745 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.962718    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs"]
Nov 02 01:32:04.982848 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.982824    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7948cc5b95-b87qg"]
Nov 02 01:32:04.986204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:04.986179    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn"]
Nov 02 01:32:04.989348 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:32:04.989322    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9b8562_0baf_437e_bf85_5cc8f0e32de1.slice/crio-0936226ccf38a4e5ba2721541db2f0b90b6b3bec36670557213d8e02c9adae0f WatchSource:0}: Error finding container 0936226ccf38a4e5ba2721541db2f0b90b6b3bec36670557213d8e02c9adae0f: Status 404 returned error can't find the container with id 0936226ccf38a4e5ba2721541db2f0b90b6b3bec36670557213d8e02c9adae0f
Nov 02 01:32:05.020580 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.020557    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b8676cdd-mw579"]
Nov 02 01:32:05.023962 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:32:05.023924    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c29bb7a_6e1f_4b1a_b89f_f73cabca6774.slice/crio-ab1867bd9d74cf67445fd43393b81d6b6be285b895e58c49b220717c68391748 WatchSource:0}: Error finding container ab1867bd9d74cf67445fd43393b81d6b6be285b895e58c49b220717c68391748: Status 404 returned error can't find the container with id ab1867bd9d74cf67445fd43393b81d6b6be285b895e58c49b220717c68391748
Nov 02 01:32:05.024650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.024629    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:05.031282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.031261    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sj5db"]
Nov 02 01:32:05.034887 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.034859    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799447c5f7-7ffhs"]
Nov 02 01:32:05.038726 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:32:05.038705    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9404b1b3_624d_4246_a20a_8627cfd58e2c.slice/crio-478ac3d0d94aae09a451ca819253179f55cf04abb264ad2558dde5f80b7f3c48 WatchSource:0}: Error finding container 478ac3d0d94aae09a451ca819253179f55cf04abb264ad2558dde5f80b7f3c48: Status 404 returned error can't find the container with id 478ac3d0d94aae09a451ca819253179f55cf04abb264ad2558dde5f80b7f3c48
Nov 02 01:32:05.063657 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.063630    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799447c5f7-7ffhs"]
Nov 02 01:32:05.202008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.201977    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:05.354084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.353972    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7948cc5b95-b87qg" event={"ID":"5826852c-0e9b-4031-8213-94051d31c42c","Type":"ContainerStarted","Data":"3ffb2d706c52e1f470bc5aa32d399f8bfb5f1ae5969033697206b3d2b842ee36"}
Nov 02 01:32:05.354084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.354041    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7948cc5b95-b87qg" event={"ID":"5826852c-0e9b-4031-8213-94051d31c42c","Type":"ContainerStarted","Data":"507ea2b0e1c175845789208564d4cb777b22e1ea90be673ae61b83154ebe795a"}
Nov 02 01:32:05.355803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.355776    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b8676cdd-mw579" event={"ID":"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774","Type":"ContainerStarted","Data":"f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349"}
Nov 02 01:32:05.355958 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.355821    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b8676cdd-mw579" event={"ID":"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774","Type":"ContainerStarted","Data":"ab1867bd9d74cf67445fd43393b81d6b6be285b895e58c49b220717c68391748"}
Nov 02 01:32:05.355958 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.355886    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:32:05.357393 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.357369    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs" event={"ID":"944b4b8f-a639-4153-a4f7-3dc0e5691225","Type":"ContainerStarted","Data":"3f86fc17f50c1f3516d8c2a71a34345d5243c8422c767c43ca2bcfe9717380ac"}
Nov 02 01:32:05.358740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.358717    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8" event={"ID":"26929d97-925d-4839-9bd7-e0fc4b3373a0","Type":"ContainerStarted","Data":"0c68f0723fa9575a61e22a310c9d8c254a48a1e3ca12a0546e9976996751f85c"}
Nov 02 01:32:05.360257 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.360231    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sj5db" event={"ID":"4161f42e-a6d1-48ea-b5dc-4d21e16facf6","Type":"ContainerStarted","Data":"210fe3e184a9ce8fa840487514185a2da8eb995c334516e27f26627c5785f7e3"}
Nov 02 01:32:05.361703 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.361680    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn" event={"ID":"ce9b8562-0baf-437e-bf85-5cc8f0e32de1","Type":"ContainerStarted","Data":"0936226ccf38a4e5ba2721541db2f0b90b6b3bec36670557213d8e02c9adae0f"}
Nov 02 01:32:05.363487 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.363464    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs" event={"ID":"9404b1b3-624d-4246-a20a-8627cfd58e2c","Type":"ContainerStarted","Data":"74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7"}
Nov 02 01:32:05.363593 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.363502    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs" event={"ID":"9404b1b3-624d-4246-a20a-8627cfd58e2c","Type":"ContainerStarted","Data":"478ac3d0d94aae09a451ca819253179f55cf04abb264ad2558dde5f80b7f3c48"}
Nov 02 01:32:05.363652 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.363611    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:32:05.374947 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.374917    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:05.521721 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.521689    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:32:05.524184 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.524159    4912 patch_prober.go:29] interesting pod/router-default-7948cc5b95-b87qg container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Nov 02 01:32:05.524184 ip-10-0-135-33 kubenswrapper[4912]: [+]has-synced ok
Nov 02 01:32:05.524184 ip-10-0-135-33 kubenswrapper[4912]: [+]process-running ok
Nov 02 01:32:05.524184 ip-10-0-135-33 kubenswrapper[4912]: healthz check failed
Nov 02 01:32:05.524400 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.524213    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-7948cc5b95-b87qg" podUID="5826852c-0e9b-4031-8213-94051d31c42c" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 02 01:32:05.563471 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.563441    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:05.764374 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:05.764333    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:06.523676 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:06.523644    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:32:07.378809 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:07.378778    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qxgb8" event={"ID":"26929d97-925d-4839-9bd7-e0fc4b3373a0","Type":"ContainerStarted","Data":"d2b75ca2674de0f92a8aa2232ec8635a3a51a3464b685f6ce6fadc23897ea6cd"}
Nov 02 01:32:07.379257 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:07.378824    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:32:07.380491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:07.380465    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7948cc5b95-b87qg"
Nov 02 01:32:08.384111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:08.384078    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sj5db" event={"ID":"4161f42e-a6d1-48ea-b5dc-4d21e16facf6","Type":"ContainerStarted","Data":"083cfa306751dd83a514f62b7216615f63318195ae86f072a97883ab0be8a7fc"}
Nov 02 01:32:08.384518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:08.384129    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sj5db" event={"ID":"4161f42e-a6d1-48ea-b5dc-4d21e16facf6","Type":"ContainerStarted","Data":"4653c11cddc4d8c7b2a6bb938bb9574e57288912759bb753cf09b174f72c41f2"}
Nov 02 01:32:08.384518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:08.384206    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sj5db"
Nov 02 01:32:08.385787 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:08.385763    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gbxrn" event={"ID":"ce9b8562-0baf-437e-bf85-5cc8f0e32de1","Type":"ContainerStarted","Data":"5e0cfa0ac974a9d921c32baa5ba0fc85ff50514872d7ef4fe1c65b071705d20a"}
Nov 02 01:32:08.387620 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:08.387594    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs" event={"ID":"944b4b8f-a639-4153-a4f7-3dc0e5691225","Type":"ContainerStarted","Data":"fa802aea97f98dca30ba4cdf9e37f635461b66f8a110939093c722159aa4ee97"}
Nov 02 01:32:08.387755 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:08.387636    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gbhrs" event={"ID":"944b4b8f-a639-4153-a4f7-3dc0e5691225","Type":"ContainerStarted","Data":"dc976cfdec7787df7fb11f57fa447334cf990f8388da0bc398d995a26372c002"}
Nov 02 01:32:08.858295 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:08.858216    4912 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:32:08.858672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:08.858659    4912 scope.go:122] "RemoveContainer" containerID="8877f603d5915124dcf14b371a01e0cf58bdd0d5e059b2fee2693152ca410355"
Nov 02 01:32:08.858876 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:32:08.858856    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m8ck_openshift-console-operator(277a3a65-7e29-4cb7-a913-d364040935b0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0"
Nov 02 01:32:08.993633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:08.993602    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:09.219876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:09.219841    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:09.401252 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:09.401224    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:09.583464 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:09.583361    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:10.056960 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:10.056928    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:10.275481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:10.275441    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:12.137466 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:32:12.137431    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:32:13.327229 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:13.327200    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:13.553406 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:13.553373    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:13.740876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:13.740840    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:18.393449 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:18.393418    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sj5db"
Nov 02 01:32:18.884068 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:18.884016    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:19.279341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:19.279271    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hqwck"
Nov 02 01:32:19.560854 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:19.560767    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:19.844560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:19.844476    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:20.063736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.063697    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:20.242330 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.242288    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:20.376546 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.376516    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zvzz9"]
Nov 02 01:32:20.418356 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.418321    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:20.420666 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.420645    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.525437 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.525363    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 01:32:20.525437 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.525417    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 01:32:20.525605 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.525464    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6vdkx\""
Nov 02 01:32:20.527818 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.527801    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 01:32:20.565351 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.565313    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 01:32:20.580350 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.580320    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6nz\" (UniqueName: \"kubernetes.io/projected/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-kube-api-access-9s6nz\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.580471 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.580355    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-metrics-client-ca\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.580471 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.580393    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-wtmp\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.580471 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.580465    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-textfile\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.580578 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.580560    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-sys\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.580624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.580610    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.580701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.580636    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-tls\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.580701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.580656    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-root\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.580701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.580696    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.601136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.601104    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:20.681512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681484    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-wtmp\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.681512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681515    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-textfile\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.681719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681542    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-sys\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.681719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681594    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-sys\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.681719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681613    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.681719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681657    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-wtmp\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.681719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681667    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-tls\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.681719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681697    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-root\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.682015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681732    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.682015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681776    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6nz\" (UniqueName: \"kubernetes.io/projected/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-kube-api-access-9s6nz\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.682015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681810    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-metrics-client-ca\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.682015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681854    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-textfile\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.682015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.681866    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-root\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.682304 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.682240    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.682720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.682699    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-metrics-client-ca\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.684389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.684352    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.684527 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.684436    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-node-exporter-tls\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.742482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.742459    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6nz\" (UniqueName: \"kubernetes.io/projected/b938abbc-9101-4d8e-9f4e-f07e99ddc3a2-kube-api-access-9s6nz\") pod \"node-exporter-zvzz9\" (UID: \"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2\") " pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.771433 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.771411    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zvzz9"
Nov 02 01:32:20.787093 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.787009    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:20.803104 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:32:20.803079    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb938abbc_9101_4d8e_9f4e_f07e99ddc3a2.slice/crio-1378df89ce4fe2818330f923f6c55039d55b536588a8e195e9b88cf8ceedc18a WatchSource:0}: Error finding container 1378df89ce4fe2818330f923f6c55039d55b536588a8e195e9b88cf8ceedc18a: Status 404 returned error can't find the container with id 1378df89ce4fe2818330f923f6c55039d55b536588a8e195e9b88cf8ceedc18a
Nov 02 01:32:20.963401 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:20.963362    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:21.218443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:21.218410    4912 scope.go:122] "RemoveContainer" containerID="8877f603d5915124dcf14b371a01e0cf58bdd0d5e059b2fee2693152ca410355"
Nov 02 01:32:21.432087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:21.432028    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvzz9" event={"ID":"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2","Type":"ContainerStarted","Data":"1378df89ce4fe2818330f923f6c55039d55b536588a8e195e9b88cf8ceedc18a"}
Nov 02 01:32:21.433923 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:21.433903    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:32:21.434067 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:21.433947    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" event={"ID":"277a3a65-7e29-4cb7-a913-d364040935b0","Type":"ContainerStarted","Data":"dbb595a0230d881024992bfd84504597e01da7a3f443388c958cb3628fbaa44a"}
Nov 02 01:32:21.434247 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:21.434230    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:32:21.435157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:21.435135    4912 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-2m8ck container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body=
Nov 02 01:32:21.435236 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:21.435172    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck" podUID="277a3a65-7e29-4cb7-a913-d364040935b0" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused"
Nov 02 01:32:21.569444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:21.569360    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:21.753998 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:21.753964    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:22.424426 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:22.424394    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:22.443584 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:22.443555    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m8ck"
Nov 02 01:32:23.002040 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:23.002011    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:23.311628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:23.311579    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:23.443788 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:23.443749    4912 generic.go:356] "Generic (PLEG): container finished" podID="b938abbc-9101-4d8e-9f4e-f07e99ddc3a2" containerID="d5b1bfac811749387b3b6cd50b900cfb5573c38e56be561a8d78a4b34e45d8c8" exitCode=0
Nov 02 01:32:23.443962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:23.443840    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvzz9" event={"ID":"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2","Type":"ContainerDied","Data":"d5b1bfac811749387b3b6cd50b900cfb5573c38e56be561a8d78a4b34e45d8c8"}
Nov 02 01:32:23.492039 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:23.492008    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:23.827163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:23.827082    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:24.099156 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:24.099070    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:24.274751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:24.274722    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:24.450648 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:24.450611    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvzz9" event={"ID":"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2","Type":"ContainerStarted","Data":"c9c9e656180dbf0e35003a47d56e3592e4f5097fc10315478e2309edd25277be"}
Nov 02 01:32:24.450853 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:24.450672    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvzz9" event={"ID":"b938abbc-9101-4d8e-9f4e-f07e99ddc3a2","Type":"ContainerStarted","Data":"de1482de9b03b43781874dab7c37063ffbbb4e73e2648558a6a40cbeccdff7fe"}
Nov 02 01:32:24.452712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:24.452687    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:24.568916 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:24.568859    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zvzz9" podStartSLOduration=2.164021642 podStartE2EDuration="4.5688441s" totalImagesPullingTime="2.317315169s" totalInitContainerRuntime="87.507289ms" isStatefulPod=true podCreationTimestamp="2026-11-02 01:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:24.568434644 +0000 UTC m=+37.473900088" watchObservedRunningTime="2026-11-02 01:32:24.5688441 +0000 UTC m=+37.474309499"
Nov 02 01:32:24.659575 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:24.659535    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:24.835954 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:24.835879    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:25.013000 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:25.012945    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:25.190313 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:25.190278    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:25.361614 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:25.361587    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:32:25.369155 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:25.369134    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:32:27.689728 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:27.689640    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:27.918514 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:27.918482    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:28.403254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:28.403223    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:28.573518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:28.573473    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:29.567423 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:29.567384    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:30.378583 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.378527    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57b8676cdd-mw579" podUID="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" containerName="registry" containerID="cri-o://f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349" gracePeriod=30
Nov 02 01:32:30.390569 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.390533    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs" podUID="9404b1b3-624d-4246-a20a-8627cfd58e2c" containerName="registry" containerID="cri-o://74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7" gracePeriod=30
Nov 02 01:32:30.630387 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.630340    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:32:30.633254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.633234    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:32:30.660990 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.660969    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates\") pod \"9404b1b3-624d-4246-a20a-8627cfd58e2c\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") "
Nov 02 01:32:30.661148 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.660999    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration\") pod \"9404b1b3-624d-4246-a20a-8627cfd58e2c\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") "
Nov 02 01:32:30.661148 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661021    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token\") pod \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") "
Nov 02 01:32:30.661148 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661040    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted\") pod \"9404b1b3-624d-4246-a20a-8627cfd58e2c\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") "
Nov 02 01:32:30.661328 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661269    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") pod \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") "
Nov 02 01:32:30.661512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661468    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates\") pod \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") "
Nov 02 01:32:30.661586 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661543    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") pod \"9404b1b3-624d-4246-a20a-8627cfd58e2c\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") "
Nov 02 01:32:30.661625 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661592    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token\") pod \"9404b1b3-624d-4246-a20a-8627cfd58e2c\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") "
Nov 02 01:32:30.661677 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661626    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates" pod "9404b1b3-624d-4246-a20a-8627cfd58e2c" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:30.661737 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661713    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration\") pod \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") "
Nov 02 01:32:30.661801 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661750    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz\") pod \"9404b1b3-624d-4246-a20a-8627cfd58e2c\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") "
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661875    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted\") pod \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") "
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661961    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca\") pod \"9404b1b3-624d-4246-a20a-8627cfd58e2c\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") "
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.661987    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz\") pod \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") "
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.662028    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets\") pod \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") "
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.662083    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets\") pod \"9404b1b3-624d-4246-a20a-8627cfd58e2c\" (UID: \"9404b1b3-624d-4246-a20a-8627cfd58e2c\") "
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.662109    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca\") pod \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\" (UID: \"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774\") "
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.662428    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates" pod "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.662720    4912 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-certificates\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.662752    4912 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-certificates\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.663370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.663045    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca" pod "9404b1b3-624d-4246-a20a-8627cfd58e2c" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:30.665018 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.664988    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca" pod "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:30.665751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.665701    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls" pod "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:30.671540 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.671512    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets" pod "9404b1b3-624d-4246-a20a-8627cfd58e2c" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:30.671716 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.671685    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token" pod "9404b1b3-624d-4246-a20a-8627cfd58e2c" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:30.672651 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.672622    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls" pod "9404b1b3-624d-4246-a20a-8627cfd58e2c" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:30.672765 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.672679    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration" pod "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:30.672961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.672914    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token" pod "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:30.673371 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.673346    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz" pod "9404b1b3-624d-4246-a20a-8627cfd58e2c" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c"). InnerVolumeSpecName "kube-api-access-jjcgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:30.673525 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.673432    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets" pod "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:30.673596 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.673552    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration" pod "9404b1b3-624d-4246-a20a-8627cfd58e2c" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:30.673687 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.673643    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz" pod "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774"). InnerVolumeSpecName "kube-api-access-x7ppz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:30.674956 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.674932    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted" pod "9404b1b3-624d-4246-a20a-8627cfd58e2c" (UID: "9404b1b3-624d-4246-a20a-8627cfd58e2c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:32:30.677238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.677217    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted" pod "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" (UID: "6c29bb7a-6e1f-4b1a-b89f-f73cabca6774"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:32:30.763856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763825    4912 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9404b1b3-624d-4246-a20a-8627cfd58e2c-trusted-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.763856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763851    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ppz\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-kube-api-access-x7ppz\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.763856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763863    4912 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-installation-pull-secrets\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763872    4912 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-installation-pull-secrets\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763881    4912 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-trusted-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763890    4912 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9404b1b3-624d-4246-a20a-8627cfd58e2c-image-registry-private-configuration\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763899    4912 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-bound-sa-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763908    4912 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9404b1b3-624d-4246-a20a-8627cfd58e2c-ca-trust-extracted\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763917    4912 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-registry-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763925    4912 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-registry-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763935    4912 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-bound-sa-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763943    4912 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-image-registry-private-configuration\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763952    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjcgz\" (UniqueName: \"kubernetes.io/projected/9404b1b3-624d-4246-a20a-8627cfd58e2c-kube-api-access-jjcgz\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:30.764107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:30.763961    4912 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774-ca-trust-extracted\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:31.481172 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.481089    4912 generic.go:356] "Generic (PLEG): container finished" podID="9404b1b3-624d-4246-a20a-8627cfd58e2c" containerID="74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7" exitCode=0
Nov 02 01:32:31.481172 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.481143    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs"
Nov 02 01:32:31.481386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.481170    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs" event={"ID":"9404b1b3-624d-4246-a20a-8627cfd58e2c","Type":"ContainerDied","Data":"74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7"}
Nov 02 01:32:31.481386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.481229    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799447c5f7-7ffhs" event={"ID":"9404b1b3-624d-4246-a20a-8627cfd58e2c","Type":"ContainerDied","Data":"478ac3d0d94aae09a451ca819253179f55cf04abb264ad2558dde5f80b7f3c48"}
Nov 02 01:32:31.481386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.481259    4912 scope.go:122] "RemoveContainer" containerID="74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7"
Nov 02 01:32:31.482767 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.482633    4912 generic.go:356] "Generic (PLEG): container finished" podID="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" containerID="f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349" exitCode=0
Nov 02 01:32:31.482767 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.482697    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b8676cdd-mw579" event={"ID":"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774","Type":"ContainerDied","Data":"f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349"}
Nov 02 01:32:31.482767 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.482707    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b8676cdd-mw579"
Nov 02 01:32:31.482767 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.482731    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b8676cdd-mw579" event={"ID":"6c29bb7a-6e1f-4b1a-b89f-f73cabca6774","Type":"ContainerDied","Data":"ab1867bd9d74cf67445fd43393b81d6b6be285b895e58c49b220717c68391748"}
Nov 02 01:32:31.489894 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.489877    4912 scope.go:122] "RemoveContainer" containerID="74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7"
Nov 02 01:32:31.490186 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:32:31.490163    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7\": container with ID starting with 74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7 not found: ID does not exist" containerID="74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7"
Nov 02 01:32:31.490268 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.490198    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7"} err="failed to get container status \"74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7\": rpc error: code = NotFound desc = could not find container \"74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7\": container with ID starting with 74adfb325981740c0accaca25110d0ac0635936ae5df468a0a56011579cb17e7 not found: ID does not exist"
Nov 02 01:32:31.490268 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.490236    4912 scope.go:122] "RemoveContainer" containerID="f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349"
Nov 02 01:32:31.497550 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.497535    4912 scope.go:122] "RemoveContainer" containerID="f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349"
Nov 02 01:32:31.497780 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:32:31.497763    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349\": container with ID starting with f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349 not found: ID does not exist" containerID="f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349"
Nov 02 01:32:31.497837 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.497787    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349"} err="failed to get container status \"f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349\": rpc error: code = NotFound desc = could not find container \"f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349\": container with ID starting with f4b298cb117b105d342e61db6b4fb6e53fef03476c6f9316ae7ccfd5aaffd349 not found: ID does not exist"
Nov 02 01:32:31.694116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.694080    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799447c5f7-7ffhs"]
Nov 02 01:32:31.743460 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.743383    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-799447c5f7-7ffhs"]
Nov 02 01:32:31.808889 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.808860    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b8676cdd-mw579"]
Nov 02 01:32:31.818322 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.818298    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57b8676cdd-mw579"]
Nov 02 01:32:31.849316 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:31.849286    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:32.021724 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:32.021636    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:32.199405 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:32.199374    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:32.404253 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:32.404224    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:33.222905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:33.222873    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" path="/var/lib/kubelet/pods/6c29bb7a-6e1f-4b1a-b89f-f73cabca6774/volumes"
Nov 02 01:32:33.223361 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:33.223346    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9404b1b3-624d-4246-a20a-8627cfd58e2c" path="/var/lib/kubelet/pods/9404b1b3-624d-4246-a20a-8627cfd58e2c/volumes"
Nov 02 01:32:34.407036 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:34.406995    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:35.581824 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:35.581792    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:35.757176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:35.757145    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:37.216367 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:37.216318    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:37.443751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:37.443714    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:37.619812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:37.619777    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:37.820202 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:37.820163    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:39.103803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:39.103769    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:39.273174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:39.273134    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:40.614923 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:40.614893    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:40.802271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:40.802242    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:42.136435 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:32:42.136394    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:32:46.401584 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:46.401552    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:46.583644 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:46.583607    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:50.175311 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:50.175277    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:50.344480 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:50.344448    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:54.026544 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:54.026508    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:54.212239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:54.212208    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:54.396341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:54.396305    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:58.414826 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:58.414787    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:58.595970 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:58.595935    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:59.588127 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:59.588029    4912 generic.go:356] "Generic (PLEG): container finished" podID="8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4" containerID="aac04af2ee54f1a2da1f739fdcf76f918f47ef4eab69dfea106d6f64ef59e02d" exitCode=0
Nov 02 01:32:59.588127 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:59.588097    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw" event={"ID":"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4","Type":"ContainerDied","Data":"aac04af2ee54f1a2da1f739fdcf76f918f47ef4eab69dfea106d6f64ef59e02d"}
Nov 02 01:32:59.588550 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:59.588410    4912 scope.go:122] "RemoveContainer" containerID="aac04af2ee54f1a2da1f739fdcf76f918f47ef4eab69dfea106d6f64ef59e02d"
Nov 02 01:32:59.788488 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:59.788447    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:59.984579 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:32:59.984544    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:00.593366 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:00.593331    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xm8rw" event={"ID":"8f4b4cf3-6159-4ce5-8f7b-abeb3377bcf4","Type":"ContainerStarted","Data":"5e593dcc9eeab47f61abed03ef5713a9bbabdf241c2280897bb67c3ab3e46d96"}
Nov 02 01:33:04.609105 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:04.609041    4912 generic.go:356] "Generic (PLEG): container finished" podID="5c414b43-ca48-4f3e-a38a-c82f2cc27283" containerID="ebc87cfb861305311296a3adfcf5d35611ab8d871d179b738972eb12f63dc6b9" exitCode=0
Nov 02 01:33:04.609105 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:04.609103    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv" event={"ID":"5c414b43-ca48-4f3e-a38a-c82f2cc27283","Type":"ContainerDied","Data":"ebc87cfb861305311296a3adfcf5d35611ab8d871d179b738972eb12f63dc6b9"}
Nov 02 01:33:04.609654 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:04.609505    4912 scope.go:122] "RemoveContainer" containerID="ebc87cfb861305311296a3adfcf5d35611ab8d871d179b738972eb12f63dc6b9"
Nov 02 01:33:05.614708 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:05.614678    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4zktv" event={"ID":"5c414b43-ca48-4f3e-a38a-c82f2cc27283","Type":"ContainerStarted","Data":"da8304a1b1467fc49e8daf3db0dc958ae270f29124249c16fbfe833d621247b4"}
Nov 02 01:33:07.624911 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:07.624881    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:07.800693 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:07.800659    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:12.135790 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:33:12.135754    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:33:18.662186 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:18.662151    4912 generic.go:356] "Generic (PLEG): container finished" podID="c1284f23-412e-4a65-970e-25ec56f3da4a" containerID="98ba3ff976c768ca86ad5fa72e0f149cb94550654e9bc4c3c1f5743a62803514" exitCode=0
Nov 02 01:33:18.662620 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:18.662206    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bv7fm" event={"ID":"c1284f23-412e-4a65-970e-25ec56f3da4a","Type":"ContainerDied","Data":"98ba3ff976c768ca86ad5fa72e0f149cb94550654e9bc4c3c1f5743a62803514"}
Nov 02 01:33:18.662620 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:18.662497    4912 scope.go:122] "RemoveContainer" containerID="98ba3ff976c768ca86ad5fa72e0f149cb94550654e9bc4c3c1f5743a62803514"
Nov 02 01:33:19.667618 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:19.667578    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bv7fm" event={"ID":"c1284f23-412e-4a65-970e-25ec56f3da4a","Type":"ContainerStarted","Data":"d195c16bb29c9e0fca2c913e3b37b6e5a103226aea1a88f55764e5712ca9fa46"}
Nov 02 01:33:24.113384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:24.113350    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:24.283193 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:24.283166    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:37.973507 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:37.973470    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:42.136290 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:33:42.136251    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:33:57.987727 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:33:57.987646    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:01.881910 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:01.881879    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:02.057571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:02.057534    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:05.073021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:05.072983    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:05.306393 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:05.306360    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:05.479837 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:05.479806    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:12.136501 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:34:12.136457    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:34:14.533528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:14.533481    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:14.714607 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:14.714570    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:24.118527 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:24.118470    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:24.340133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:24.340098    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:38.315931 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:34:38.315895    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:42.136155 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:34:42.136117    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:35:03.497027 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:35:03.496993    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:12.136123 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:35:12.136081    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:35:38.669894 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:35:38.669808    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:42.137224 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:35:42.137185    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:36:12.136220 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:36:12.136182    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:36:28.077976 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:28.077934    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:29.965945 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:29.965912    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:30.613893 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.613859    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"]
Nov 02 01:36:30.614178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.614165    4912 cpu_manager.go:414] "containerMap: removing container" podUID="9404b1b3-624d-4246-a20a-8627cfd58e2c" containerName="registry"
Nov 02 01:36:30.614228 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.614178    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9404b1b3-624d-4246-a20a-8627cfd58e2c" containerName="registry"
Nov 02 01:36:30.614228 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.614198    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" containerName="registry"
Nov 02 01:36:30.614228 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.614204    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" containerName="registry"
Nov 02 01:36:30.614325 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.614261    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="9404b1b3-624d-4246-a20a-8627cfd58e2c" containerName="registry"
Nov 02 01:36:30.614325 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.614271    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c29bb7a-6e1f-4b1a-b89f-f73cabca6774" containerName="registry"
Nov 02 01:36:30.617260 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.617244    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.669133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.669101    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 01:36:30.718691 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.718654    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 01:36:30.718872 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.718781    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-zztzm\""
Nov 02 01:36:30.719758 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.719730    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"]
Nov 02 01:36:30.720480 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.720463    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"]
Nov 02 01:36:30.723719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.723704    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:30.772735 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.772704    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"]
Nov 02 01:36:30.783388 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.783362    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.783574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.783395    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.783574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.783436    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbtn\" (UniqueName: \"kubernetes.io/projected/fef8b8f9-2024-4b46-95f0-7f711910f3d1-kube-api-access-zzbtn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.800497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.800474    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-5xxfn\""
Nov 02 01:36:30.815987 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.815959    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"]
Nov 02 01:36:30.819246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.819231    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:30.871007 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.870980    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-w4msw\""
Nov 02 01:36:30.878452 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.878426    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"]
Nov 02 01:36:30.884435 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.884381    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47sf2\" (UniqueName: \"kubernetes.io/projected/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-kube-api-access-47sf2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:30.884435 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.884433    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.884695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.884542    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:30.884695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.884582    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:30.884695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.884617    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.884695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.884655    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbtn\" (UniqueName: \"kubernetes.io/projected/fef8b8f9-2024-4b46-95f0-7f711910f3d1-kube-api-access-zzbtn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.885446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.885212    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.885446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.885317    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.917216 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.917185    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"]
Nov 02 01:36:30.920603 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.920587    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:30.927265 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.927243    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:30.945442 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.945414    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbtn\" (UniqueName: \"kubernetes.io/projected/fef8b8f9-2024-4b46-95f0-7f711910f3d1-kube-api-access-zzbtn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:30.972675 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.972648    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-8n6f7\""
Nov 02 01:36:30.978357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.978330    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"]
Nov 02 01:36:30.985373 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.985346    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:30.985489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.985389    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll5dk\" (UniqueName: \"kubernetes.io/projected/7000ca06-b94f-460d-a298-c7fe79254edd-kube-api-access-ll5dk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:30.985565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.985483    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47sf2\" (UniqueName: \"kubernetes.io/projected/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-kube-api-access-47sf2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:30.985565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.985529    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:30.985565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.985556    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:30.985723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.985643    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:30.985917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.985895    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:30.986024 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:30.986003    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:31.046462 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.046416    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47sf2\" (UniqueName: \"kubernetes.io/projected/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-kube-api-access-47sf2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:31.086374 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.086343    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.086581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.086385    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6v5\" (UniqueName: \"kubernetes.io/projected/be7b7a96-bbac-469f-8320-f1aed5f1f787-kube-api-access-zp6v5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.086581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.086426    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:31.086581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.086495    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.086581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.086568    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:31.086961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.086597    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll5dk\" (UniqueName: \"kubernetes.io/projected/7000ca06-b94f-460d-a298-c7fe79254edd-kube-api-access-ll5dk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:31.086961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.086828    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:31.086961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.086869    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:31.147652 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.147579    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll5dk\" (UniqueName: \"kubernetes.io/projected/7000ca06-b94f-460d-a298-c7fe79254edd-kube-api-access-ll5dk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:31.187346 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.187308    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.187518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.187405    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.187518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.187432    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6v5\" (UniqueName: \"kubernetes.io/projected/be7b7a96-bbac-469f-8320-f1aed5f1f787-kube-api-access-zp6v5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.187735 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.187713    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.187780 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.187746    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.233410 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.233378    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:36:31.250839 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.250809    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6v5\" (UniqueName: \"kubernetes.io/projected/be7b7a96-bbac-469f-8320-f1aed5f1f787-kube-api-access-zp6v5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.292618 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.292585    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"]
Nov 02 01:36:31.301106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.300829    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.339166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.339142    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:36:31.353934 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.353843    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-lm57z\""
Nov 02 01:36:31.355002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.354977    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"]
Nov 02 01:36:31.434171 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.434103    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:36:31.490708 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.490676    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.490846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.490730    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8h5\" (UniqueName: \"kubernetes.io/projected/14b25417-19b5-4ad0-b6e3-5ec163320330-kube-api-access-sg8h5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.490846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.490813    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.536539 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.536507    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:36:31.591574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.591545    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.591705 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.591627    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.591705 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.591662    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8h5\" (UniqueName: \"kubernetes.io/projected/14b25417-19b5-4ad0-b6e3-5ec163320330-kube-api-access-sg8h5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.592435 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.592394    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.592435 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.592402    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.608045 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.608020    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"]
Nov 02 01:36:31.611589 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:31.611561    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef8b8f9_2024_4b46_95f0_7f711910f3d1.slice/crio-a523c5a0d6bdcd24a2b1d6e6565950a8e17c05e12768506125e9ed519c8df677 WatchSource:0}: Error finding container a523c5a0d6bdcd24a2b1d6e6565950a8e17c05e12768506125e9ed519c8df677: Status 404 returned error can't find the container with id a523c5a0d6bdcd24a2b1d6e6565950a8e17c05e12768506125e9ed519c8df677
Nov 02 01:36:31.652813 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.652782    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8h5\" (UniqueName: \"kubernetes.io/projected/14b25417-19b5-4ad0-b6e3-5ec163320330-kube-api-access-sg8h5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:31.713217 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.713192    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"]
Nov 02 01:36:31.715477 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:31.715450    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf2ff2c_64d1_44c9_949f_8f3c26ac673a.slice/crio-c82cdd6770a8be0f455f377e25171e9629a86bbbd951da1d215bbe0098153d9a WatchSource:0}: Error finding container c82cdd6770a8be0f455f377e25171e9629a86bbbd951da1d215bbe0098153d9a: Status 404 returned error can't find the container with id c82cdd6770a8be0f455f377e25171e9629a86bbbd951da1d215bbe0098153d9a
Nov 02 01:36:31.806189 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.806104    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"]
Nov 02 01:36:31.808333 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:31.808307    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7000ca06_b94f_460d_a298_c7fe79254edd.slice/crio-c290c4d748c4e3562d25ad1aa6625f40306ed77129a87a240c4cbb78c585be45 WatchSource:0}: Error finding container c290c4d748c4e3562d25ad1aa6625f40306ed77129a87a240c4cbb78c585be45: Status 404 returned error can't find the container with id c290c4d748c4e3562d25ad1aa6625f40306ed77129a87a240c4cbb78c585be45
Nov 02 01:36:31.910236 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.910211    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"]
Nov 02 01:36:31.912794 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:31.912765    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7b7a96_bbac_469f_8320_f1aed5f1f787.slice/crio-de86dfa7b5e56d3a10f13d0d9c7e9f62d0e32fe8284ed65bb9b1a21553313ef6 WatchSource:0}: Error finding container de86dfa7b5e56d3a10f13d0d9c7e9f62d0e32fe8284ed65bb9b1a21553313ef6: Status 404 returned error can't find the container with id de86dfa7b5e56d3a10f13d0d9c7e9f62d0e32fe8284ed65bb9b1a21553313ef6
Nov 02 01:36:31.922773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:31.922753    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:36:32.299389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:32.299313    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"]
Nov 02 01:36:32.302624 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:32.302595    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b25417_19b5_4ad0_b6e3_5ec163320330.slice/crio-1b296e346f61b823e8896b44998cfe365c63f42b0864dfdfadd352898536a1e6 WatchSource:0}: Error finding container 1b296e346f61b823e8896b44998cfe365c63f42b0864dfdfadd352898536a1e6: Status 404 returned error can't find the container with id 1b296e346f61b823e8896b44998cfe365c63f42b0864dfdfadd352898536a1e6
Nov 02 01:36:32.334950 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:32.334907    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z" event={"ID":"14b25417-19b5-4ad0-b6e3-5ec163320330","Type":"ContainerStarted","Data":"1b296e346f61b823e8896b44998cfe365c63f42b0864dfdfadd352898536a1e6"}
Nov 02 01:36:32.336145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:32.336123    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9" event={"ID":"fef8b8f9-2024-4b46-95f0-7f711910f3d1","Type":"ContainerStarted","Data":"a523c5a0d6bdcd24a2b1d6e6565950a8e17c05e12768506125e9ed519c8df677"}
Nov 02 01:36:32.337336 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:32.337309    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl" event={"ID":"be7b7a96-bbac-469f-8320-f1aed5f1f787","Type":"ContainerStarted","Data":"de86dfa7b5e56d3a10f13d0d9c7e9f62d0e32fe8284ed65bb9b1a21553313ef6"}
Nov 02 01:36:32.338374 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:32.338353    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9" event={"ID":"baf2ff2c-64d1-44c9-949f-8f3c26ac673a","Type":"ContainerStarted","Data":"c82cdd6770a8be0f455f377e25171e9629a86bbbd951da1d215bbe0098153d9a"}
Nov 02 01:36:32.339370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:32.339352    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97" event={"ID":"7000ca06-b94f-460d-a298-c7fe79254edd","Type":"ContainerStarted","Data":"c290c4d748c4e3562d25ad1aa6625f40306ed77129a87a240c4cbb78c585be45"}
Nov 02 01:36:32.897857 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:32.897816    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"]
Nov 02 01:36:32.901498 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:32.901475    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.005592 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.005534    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxvk\" (UniqueName: \"kubernetes.io/projected/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-kube-api-access-bgxvk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.005740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.005609    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.005740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.005626    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-wrh6r\""
Nov 02 01:36:33.005740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.005641    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.007713 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.007686    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"]
Nov 02 01:36:33.106444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.106409    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxvk\" (UniqueName: \"kubernetes.io/projected/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-kube-api-access-bgxvk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.106617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.106465    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.106617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.106485    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.106889 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.106869    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.106956 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.106926    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.170071 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.169979    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxvk\" (UniqueName: \"kubernetes.io/projected/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-kube-api-access-bgxvk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.222507 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.222472    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:36:33.642880 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:33.642810    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"]
Nov 02 01:36:33.655131 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:33.653952    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c6a022_03df_4bb4_9d2b_8adeaa3d8d65.slice/crio-33cc8fff3502507a243f3443b22b27b6a8fb48b454b88ca88e6a2b06a8bc4364 WatchSource:0}: Error finding container 33cc8fff3502507a243f3443b22b27b6a8fb48b454b88ca88e6a2b06a8bc4364: Status 404 returned error can't find the container with id 33cc8fff3502507a243f3443b22b27b6a8fb48b454b88ca88e6a2b06a8bc4364
Nov 02 01:36:34.355922 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:34.355883    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq" event={"ID":"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65","Type":"ContainerStarted","Data":"33cc8fff3502507a243f3443b22b27b6a8fb48b454b88ca88e6a2b06a8bc4364"}
Nov 02 01:36:35.448853 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.446864    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"]
Nov 02 01:36:35.450937 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.450912    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.531946 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.531904    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.532142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.532017    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.532142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.532050    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnf5c\" (UniqueName: \"kubernetes.io/projected/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-kube-api-access-gnf5c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.552771 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.552736    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-8tkpx\""
Nov 02 01:36:35.554071 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.554028    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"]
Nov 02 01:36:35.633136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.633103    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.633332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.633153    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnf5c\" (UniqueName: \"kubernetes.io/projected/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-kube-api-access-gnf5c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.633332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.633212    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.633926 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.633897    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.634248 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.634185    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.702951 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.702877    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnf5c\" (UniqueName: \"kubernetes.io/projected/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-kube-api-access-gnf5c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.773312 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.772773    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:36:35.876876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:35.876822    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:36.105815 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.105666    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:36.209427 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.209362    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"]
Nov 02 01:36:36.221141 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:36.221101    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a81a42e_8f2b_4377_9a2d_ad4536e2c436.slice/crio-088ec041a42dc372d84a3526afbd387608f34a68f5e43c03575a2cc867275ce6 WatchSource:0}: Error finding container 088ec041a42dc372d84a3526afbd387608f34a68f5e43c03575a2cc867275ce6: Status 404 returned error can't find the container with id 088ec041a42dc372d84a3526afbd387608f34a68f5e43c03575a2cc867275ce6
Nov 02 01:36:36.280657 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.280604    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"]
Nov 02 01:36:36.286011 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.285979    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.338542 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.338464    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58w2j\""
Nov 02 01:36:36.338542 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.338532    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 01:36:36.345501 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.345467    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"]
Nov 02 01:36:36.382814 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.382774    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st" event={"ID":"7a81a42e-8f2b-4377-9a2d-ad4536e2c436","Type":"ContainerStarted","Data":"088ec041a42dc372d84a3526afbd387608f34a68f5e43c03575a2cc867275ce6"}
Nov 02 01:36:36.411901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.411855    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:36.442366 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.442320    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.442628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.442378    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.442628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.442411    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.442628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.442433    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.442628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.442463    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79tx\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-kube-api-access-c79tx\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.442628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.442522    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.442628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.442547    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.442628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.442570    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.442628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.442598    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.543635    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.543690    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.543727    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.543760    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.543792    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79tx\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-kube-api-access-c79tx\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.543832    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.543850    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.543868    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.543889    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.544264    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.544465    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.544650    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.546072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.545242    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.548186 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.547972    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.552879 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.552746    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.555736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.555651    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.599145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.599107    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"]
Nov 02 01:36:36.604669 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.604639    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.616849 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.616115    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79tx\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-kube-api-access-c79tx\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.620979 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.618918    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nfdcd\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.632267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.632224    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:36.645335 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.645233    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.645335 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.645277    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.645565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.645347    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.645565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.645374    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.645565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.645403    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.645565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.645434    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.645565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.645465    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.645565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.645507    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.645565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.645532    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfcp\" (UniqueName: \"kubernetes.io/projected/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-kube-api-access-9pfcp\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.666106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.663992    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"]
Nov 02 01:36:36.744368 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.744338    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-6qdch\""
Nov 02 01:36:36.745976 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.745951    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.746187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.745982    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.746187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.746003    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.746187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.746035    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.746187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.746079    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.746187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.746119    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.746187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.746142    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfcp\" (UniqueName: \"kubernetes.io/projected/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-kube-api-access-9pfcp\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.746519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.746196    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.746519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.746216    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.746975 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.746951    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.747532 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.747500    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.748481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.747777    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.748481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.747954    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.748481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.748217    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.749921 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.749900    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.751838 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.751817    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.810570 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.810389    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:36.815083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.815023    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfcp\" (UniqueName: \"kubernetes.io/projected/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-kube-api-access-9pfcp\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.818706 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.818623    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58eea3b-0fb7-4fd8-98ae-415f4972c7cf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk\" (UID: \"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:36.906497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.906136    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:36.924995 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:36.924957    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:37.377704 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:37.377652    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"]
Nov 02 01:36:37.391991 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:37.391953    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c31c29_3c02_42ed_b6ef_24518ff9fa4a.slice/crio-da382ac5326c3516f8eccc4954529423fc596eeadcd855c6a017aa26e8e6011c WatchSource:0}: Error finding container da382ac5326c3516f8eccc4954529423fc596eeadcd855c6a017aa26e8e6011c: Status 404 returned error can't find the container with id da382ac5326c3516f8eccc4954529423fc596eeadcd855c6a017aa26e8e6011c
Nov 02 01:36:37.417989 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:37.417956    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58eea3b_0fb7_4fd8_98ae_415f4972c7cf.slice/crio-289530634d35072e933ad84b041e2a82caaccef7dc599314427d7710d03232d3 WatchSource:0}: Error finding container 289530634d35072e933ad84b041e2a82caaccef7dc599314427d7710d03232d3: Status 404 returned error can't find the container with id 289530634d35072e933ad84b041e2a82caaccef7dc599314427d7710d03232d3
Nov 02 01:36:37.427194 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:37.427139    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"]
Nov 02 01:36:38.411093 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.411033    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" event={"ID":"80c31c29-3c02-42ed-b6ef-24518ff9fa4a","Type":"ContainerStarted","Data":"da382ac5326c3516f8eccc4954529423fc596eeadcd855c6a017aa26e8e6011c"}
Nov 02 01:36:38.417348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.417314    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk" event={"ID":"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf","Type":"ContainerStarted","Data":"289530634d35072e933ad84b041e2a82caaccef7dc599314427d7710d03232d3"}
Nov 02 01:36:38.507340 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.507308    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"]
Nov 02 01:36:38.512509 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.512482    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.537090 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.534999    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"]
Nov 02 01:36:38.539738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.539339    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.566883 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.566842    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.567280 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.567260    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.567427 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.567408    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rw6t\" (UniqueName: \"kubernetes.io/projected/88594ff7-0c92-407f-af92-881d0aaf03f0-kube-api-access-6rw6t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.567559 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.567542    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.567693 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.567679    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.567837 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.567820    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8b2\" (UniqueName: \"kubernetes.io/projected/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-kube-api-access-2k8b2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.584129 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.584094    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"]
Nov 02 01:36:38.588385 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.588355    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"]
Nov 02 01:36:38.588550 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.588482    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.616909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.616874    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"]
Nov 02 01:36:38.622995 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.622931    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"]
Nov 02 01:36:38.623205 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.623049    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.630663 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.630630    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"]
Nov 02 01:36:38.670113 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.670019    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"]
Nov 02 01:36:38.670848 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.670813    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.670942 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.670863    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.671633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.671589    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.671722 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.671636    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq542\" (UniqueName: \"kubernetes.io/projected/99bfc103-2bb9-45ee-a6dc-77f65af421c0-kube-api-access-jq542\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.671722 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.671692    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.671813 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.671779    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfr2r\" (UniqueName: \"kubernetes.io/projected/085912b7-b205-4955-be7e-5e4a4b38f179-kube-api-access-mfr2r\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.671848 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.671808    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.672271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.672208    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.672344 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.672294    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.672344 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.672332    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rw6t\" (UniqueName: \"kubernetes.io/projected/88594ff7-0c92-407f-af92-881d0aaf03f0-kube-api-access-6rw6t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.673352 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.673294    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.673963 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.673597    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.674273 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.674255    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.674886 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.674867    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8b2\" (UniqueName: \"kubernetes.io/projected/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-kube-api-access-2k8b2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.675441 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.673941    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.675575 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.674751    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.742588 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.742517    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rw6t\" (UniqueName: \"kubernetes.io/projected/88594ff7-0c92-407f-af92-881d0aaf03f0-kube-api-access-6rw6t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.745116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.745084    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8b2\" (UniqueName: \"kubernetes.io/projected/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-kube-api-access-2k8b2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.776717    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.776837    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.776872    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq542\" (UniqueName: \"kubernetes.io/projected/99bfc103-2bb9-45ee-a6dc-77f65af421c0-kube-api-access-jq542\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.776910    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.776936    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfr2r\" (UniqueName: \"kubernetes.io/projected/085912b7-b205-4955-be7e-5e4a4b38f179-kube-api-access-mfr2r\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.776960    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.777462    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.777705    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.777985    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.778816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.778541    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.838688 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.838620    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:36:38.861048 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.860981    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfr2r\" (UniqueName: \"kubernetes.io/projected/085912b7-b205-4955-be7e-5e4a4b38f179-kube-api-access-mfr2r\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.864010 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.863954    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq542\" (UniqueName: \"kubernetes.io/projected/99bfc103-2bb9-45ee-a6dc-77f65af421c0-kube-api-access-jq542\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:38.873244 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.872871    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:36:38.945549 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.940211    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:36:38.954942 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:38.954881    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:36:39.040235 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:39.040144    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:39.323912 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:39.323827    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"]
Nov 02 01:36:39.326071 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:39.326028    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"]
Nov 02 01:36:39.377089 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:39.377030    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"]
Nov 02 01:36:39.403857 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:39.403821    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085912b7_b205_4955_be7e_5e4a4b38f179.slice/crio-0791683c688f420fee5342fb8bea6fa6fd91de82891c223e9b6f03425699207c WatchSource:0}: Error finding container 0791683c688f420fee5342fb8bea6fa6fd91de82891c223e9b6f03425699207c: Status 404 returned error can't find the container with id 0791683c688f420fee5342fb8bea6fa6fd91de82891c223e9b6f03425699207c
Nov 02 01:36:39.458279 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:39.458197    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh" event={"ID":"88594ff7-0c92-407f-af92-881d0aaf03f0","Type":"ContainerStarted","Data":"da49da6bbc14cd4f88f45e754e1002da36ec76ff4e005568a6431d909ef63efa"}
Nov 02 01:36:39.461946 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:39.461865    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7" event={"ID":"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3","Type":"ContainerStarted","Data":"cb136924a3480f5f1958421213d2cba2da52b5be6f785cf0d8462cb25edb98fb"}
Nov 02 01:36:39.469013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:39.468987    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"]
Nov 02 01:36:39.470558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:39.470523    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79" event={"ID":"085912b7-b205-4955-be7e-5e4a4b38f179","Type":"ContainerStarted","Data":"0791683c688f420fee5342fb8bea6fa6fd91de82891c223e9b6f03425699207c"}
Nov 02 01:36:39.475120 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:36:39.475073    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bfc103_2bb9_45ee_a6dc_77f65af421c0.slice/crio-afc42b6221932730155361e0e45858c1118fd331a4042997a48509cd21a209bd WatchSource:0}: Error finding container afc42b6221932730155361e0e45858c1118fd331a4042997a48509cd21a209bd: Status 404 returned error can't find the container with id afc42b6221932730155361e0e45858c1118fd331a4042997a48509cd21a209bd
Nov 02 01:36:40.322868 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:40.322823    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:40.492200 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:40.492070    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k" event={"ID":"99bfc103-2bb9-45ee-a6dc-77f65af421c0","Type":"ContainerStarted","Data":"afc42b6221932730155361e0e45858c1118fd331a4042997a48509cd21a209bd"}
Nov 02 01:36:42.180157 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:36:42.180014    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:36:47.278124 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:47.278008    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:36:47.303320 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:47.303098    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:36:47.363504 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:47.363454    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:36:47.418548 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:47.418442    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:36:49.550768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:49.550248    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:36:49.550768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:49.550334    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:36:49.550768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:49.550376    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:36:49.562459 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:49.561875    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:36:49.562459 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:49.561960    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:36:49.562459 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:49.561991    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:36:50.612219 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:50.612175    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" event={"ID":"80c31c29-3c02-42ed-b6ef-24518ff9fa4a","Type":"ContainerStarted","Data":"2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e"}
Nov 02 01:36:50.620368 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:50.620337    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk" event={"ID":"a58eea3b-0fb7-4fd8-98ae-415f4972c7cf","Type":"ContainerStarted","Data":"71fd6eeee4d7f961a35bec77eac982c553e02067294b1357ad3715ed4d05a7be"}
Nov 02 01:36:50.834700 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:50.834630    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" podStartSLOduration=2.683887783 podStartE2EDuration="14.834611861s" totalImagesPullingTime="12.150724078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:50.784695651 +0000 UTC m=+303.690161009" watchObservedRunningTime="2026-11-02 01:36:50.834611861 +0000 UTC m=+303.740077224"
Nov 02 01:36:50.907012 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:50.906968    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:50.911012 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:50.910805    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk" podStartSLOduration=2.7729966409999998 podStartE2EDuration="14.910784944s" totalImagesPullingTime="12.137788303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:50.909618373 +0000 UTC m=+303.815083730" watchObservedRunningTime="2026-11-02 01:36:50.910784944 +0000 UTC m=+303.816250302"
Nov 02 01:36:50.914421 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:50.914396    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:50.925776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:50.925749    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:50.931799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:50.931765    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:51.626490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:51.626456    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:51.627008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:51.626518    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:51.627690 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:51.627535    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:36:51.627690 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:51.627651    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk"
Nov 02 01:36:52.643781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:52.643726    4912 generic.go:356] "Generic (PLEG): container finished" podID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerID="03301fb803b72de4622b268be57052f510ecb9b88a240a22c921beb10d9b02f1" exitCode=0
Nov 02 01:36:52.644645 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:52.643961    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9" event={"ID":"baf2ff2c-64d1-44c9-949f-8f3c26ac673a","Type":"ContainerDied","Data":"03301fb803b72de4622b268be57052f510ecb9b88a240a22c921beb10d9b02f1"}
Nov 02 01:36:52.653845 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:52.653801    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9" event={"ID":"fef8b8f9-2024-4b46-95f0-7f711910f3d1","Type":"ContainerStarted","Data":"6f7cc337e39a3fa72b0c1056a1462a986b883e42d2cb809acd633e64f2854fa1"}
Nov 02 01:36:52.697072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:52.697030    4912 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:36:53.659507 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.659471    4912 generic.go:356] "Generic (PLEG): container finished" podID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerID="d9557448f96ead458c57525f309276b445b89d4071c9be0720c35f7a7bd169e2" exitCode=0
Nov 02 01:36:53.659936 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.659796    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k" event={"ID":"99bfc103-2bb9-45ee-a6dc-77f65af421c0","Type":"ContainerDied","Data":"d9557448f96ead458c57525f309276b445b89d4071c9be0720c35f7a7bd169e2"}
Nov 02 01:36:53.661864 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.661844    4912 generic.go:356] "Generic (PLEG): container finished" podID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerID="8ace9ee53d07e792b41d5615d19c015efcf872eca0694cb22eff282ca60c4283" exitCode=0
Nov 02 01:36:53.661961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.661917    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl" event={"ID":"be7b7a96-bbac-469f-8320-f1aed5f1f787","Type":"ContainerDied","Data":"8ace9ee53d07e792b41d5615d19c015efcf872eca0694cb22eff282ca60c4283"}
Nov 02 01:36:53.665198 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.665172    4912 generic.go:356] "Generic (PLEG): container finished" podID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerID="00f1908007906d16e2657a0bf64a5ba1b26f0cc899679f6c227201befc486621" exitCode=0
Nov 02 01:36:53.665312 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.665258    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7" event={"ID":"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3","Type":"ContainerDied","Data":"00f1908007906d16e2657a0bf64a5ba1b26f0cc899679f6c227201befc486621"}
Nov 02 01:36:53.667142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.667116    4912 generic.go:356] "Generic (PLEG): container finished" podID="7000ca06-b94f-460d-a298-c7fe79254edd" containerID="8dcd194fc85b7ed216b6310cba87843b47df68173a321ff24a7ae7d96e422409" exitCode=0
Nov 02 01:36:53.667249 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.667203    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97" event={"ID":"7000ca06-b94f-460d-a298-c7fe79254edd","Type":"ContainerDied","Data":"8dcd194fc85b7ed216b6310cba87843b47df68173a321ff24a7ae7d96e422409"}
Nov 02 01:36:53.669824 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.669754    4912 generic.go:356] "Generic (PLEG): container finished" podID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerID="1711193f29a72f2992893479a054b6d4b1afeca8ca3a7ac9391c15f2c06b8229" exitCode=0
Nov 02 01:36:53.669969 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.669946    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z" event={"ID":"14b25417-19b5-4ad0-b6e3-5ec163320330","Type":"ContainerDied","Data":"1711193f29a72f2992893479a054b6d4b1afeca8ca3a7ac9391c15f2c06b8229"}
Nov 02 01:36:53.672667 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.672648    4912 generic.go:356] "Generic (PLEG): container finished" podID="085912b7-b205-4955-be7e-5e4a4b38f179" containerID="24a8cf0c3c5c40b6b6e3d687ec8b49500c5f067adf06d2c5e858b551e2b4ecdb" exitCode=0
Nov 02 01:36:53.672777 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.672717    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79" event={"ID":"085912b7-b205-4955-be7e-5e4a4b38f179","Type":"ContainerDied","Data":"24a8cf0c3c5c40b6b6e3d687ec8b49500c5f067adf06d2c5e858b551e2b4ecdb"}
Nov 02 01:36:53.676640 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.676617    4912 generic.go:356] "Generic (PLEG): container finished" podID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerID="6f7cc337e39a3fa72b0c1056a1462a986b883e42d2cb809acd633e64f2854fa1" exitCode=0
Nov 02 01:36:53.676747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:53.676681    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9" event={"ID":"fef8b8f9-2024-4b46-95f0-7f711910f3d1","Type":"ContainerDied","Data":"6f7cc337e39a3fa72b0c1056a1462a986b883e42d2cb809acd633e64f2854fa1"}
Nov 02 01:36:54.683245 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:54.683044    4912 generic.go:356] "Generic (PLEG): container finished" podID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerID="1acf290af68931153b3708a578295f425f85e52208f944b386c84e460bce8533" exitCode=0
Nov 02 01:36:54.683245 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:54.683109    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh" event={"ID":"88594ff7-0c92-407f-af92-881d0aaf03f0","Type":"ContainerDied","Data":"1acf290af68931153b3708a578295f425f85e52208f944b386c84e460bce8533"}
Nov 02 01:36:54.685295 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:54.685272    4912 generic.go:356] "Generic (PLEG): container finished" podID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerID="a804fde9beb697b089eb9674063232575b73ef94ac5da034927365b9d42824c0" exitCode=0
Nov 02 01:36:54.685409 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:54.685306    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq" event={"ID":"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65","Type":"ContainerDied","Data":"a804fde9beb697b089eb9674063232575b73ef94ac5da034927365b9d42824c0"}
Nov 02 01:36:55.693395 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.693289    4912 generic.go:356] "Generic (PLEG): container finished" podID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerID="0de678006b29f3704f5c4efd55354d712558ca985353eec90141c3bdac9b5eef" exitCode=0
Nov 02 01:36:55.693395 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.693378    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st" event={"ID":"7a81a42e-8f2b-4377-9a2d-ad4536e2c436","Type":"ContainerDied","Data":"0de678006b29f3704f5c4efd55354d712558ca985353eec90141c3bdac9b5eef"}
Nov 02 01:36:55.696803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.696765    4912 generic.go:356] "Generic (PLEG): container finished" podID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerID="a108942131b074d6b6981ef3c48d2a7ae633746301bc6b22e2b55756dbe6aa7f" exitCode=0
Nov 02 01:36:55.696951 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.696917    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k" event={"ID":"99bfc103-2bb9-45ee-a6dc-77f65af421c0","Type":"ContainerDied","Data":"a108942131b074d6b6981ef3c48d2a7ae633746301bc6b22e2b55756dbe6aa7f"}
Nov 02 01:36:55.699353 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.699325    4912 generic.go:356] "Generic (PLEG): container finished" podID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerID="1389ac52d9d66454b8619b9b093f8bd0eef7e0ffd350bd73b989e02707d2f52a" exitCode=0
Nov 02 01:36:55.699498 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.699415    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl" event={"ID":"be7b7a96-bbac-469f-8320-f1aed5f1f787","Type":"ContainerDied","Data":"1389ac52d9d66454b8619b9b093f8bd0eef7e0ffd350bd73b989e02707d2f52a"}
Nov 02 01:36:55.702232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.702208    4912 generic.go:356] "Generic (PLEG): container finished" podID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerID="712664f5f375ab3932b3296ab4e46bc775006cf05faf90105539bda7b494628a" exitCode=0
Nov 02 01:36:55.702490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.702434    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9" event={"ID":"baf2ff2c-64d1-44c9-949f-8f3c26ac673a","Type":"ContainerDied","Data":"712664f5f375ab3932b3296ab4e46bc775006cf05faf90105539bda7b494628a"}
Nov 02 01:36:55.705338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.705317    4912 generic.go:356] "Generic (PLEG): container finished" podID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerID="dba3296acc1f5f594e02df341f1b2401ba81bfaa8e72ecb4500abd0bdd0c19f5" exitCode=0
Nov 02 01:36:55.705672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.705632    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7" event={"ID":"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3","Type":"ContainerDied","Data":"dba3296acc1f5f594e02df341f1b2401ba81bfaa8e72ecb4500abd0bdd0c19f5"}
Nov 02 01:36:55.708009 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.707987    4912 generic.go:356] "Generic (PLEG): container finished" podID="085912b7-b205-4955-be7e-5e4a4b38f179" containerID="054f00216d7d38d2fb40e5a3bbd29493a8589080ddf0b331449cd86c818e6969" exitCode=0
Nov 02 01:36:55.708109 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:55.708040    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79" event={"ID":"085912b7-b205-4955-be7e-5e4a4b38f179","Type":"ContainerDied","Data":"054f00216d7d38d2fb40e5a3bbd29493a8589080ddf0b331449cd86c818e6969"}
Nov 02 01:36:56.730387 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:56.730346    4912 generic.go:356] "Generic (PLEG): container finished" podID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerID="668b56988b50fba1e0f015319fc0d1293c5d36b5a07398329dfce82fd0945db2" exitCode=0
Nov 02 01:36:56.736706 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:56.731444    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st" event={"ID":"7a81a42e-8f2b-4377-9a2d-ad4536e2c436","Type":"ContainerDied","Data":"668b56988b50fba1e0f015319fc0d1293c5d36b5a07398329dfce82fd0945db2"}
Nov 02 01:36:56.746330 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:56.746290    4912 generic.go:356] "Generic (PLEG): container finished" podID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerID="d11cb1434b029e3bd659685b38785b2804bfa111aa1b1ab5cf8cf4cbae03a01d" exitCode=0
Nov 02 01:36:56.746960 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:56.746932    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh" event={"ID":"88594ff7-0c92-407f-af92-881d0aaf03f0","Type":"ContainerDied","Data":"d11cb1434b029e3bd659685b38785b2804bfa111aa1b1ab5cf8cf4cbae03a01d"}
Nov 02 01:36:56.753912 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:56.753882    4912 generic.go:356] "Generic (PLEG): container finished" podID="7000ca06-b94f-460d-a298-c7fe79254edd" containerID="44f6cb6c54f3ff4e6846a6001f4a125ae56e3ff309fc4cd49d427e486a55efa2" exitCode=0
Nov 02 01:36:56.754018 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:56.753945    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97" event={"ID":"7000ca06-b94f-460d-a298-c7fe79254edd","Type":"ContainerDied","Data":"44f6cb6c54f3ff4e6846a6001f4a125ae56e3ff309fc4cd49d427e486a55efa2"}
Nov 02 01:36:57.772416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:57.772378    4912 generic.go:356] "Generic (PLEG): container finished" podID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerID="9a19367445c99ed4549f7041babf7219cff1b12fb36cf559a9eca71518275d1d" exitCode=0
Nov 02 01:36:57.772888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:36:57.772518    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z" event={"ID":"14b25417-19b5-4ad0-b6e3-5ec163320330","Type":"ContainerDied","Data":"9a19367445c99ed4549f7041babf7219cff1b12fb36cf559a9eca71518275d1d"}
Nov 02 01:37:01.872808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:01.872768    4912 generic.go:356] "Generic (PLEG): container finished" podID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerID="5e1c9985ade54019be3a4634c82d4b67c6cd25f9449468134a182a121b2bf88b" exitCode=0
Nov 02 01:37:01.873326 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:01.872925    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9" event={"ID":"fef8b8f9-2024-4b46-95f0-7f711910f3d1","Type":"ContainerDied","Data":"5e1c9985ade54019be3a4634c82d4b67c6cd25f9449468134a182a121b2bf88b"}
Nov 02 01:37:02.904686 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:02.904630    4912 generic.go:356] "Generic (PLEG): container finished" podID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerID="52fe236418e365c85ef771811dd2f05357c2473f31991a49ff8af4432a8c8563" exitCode=0
Nov 02 01:37:02.905338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:02.904699    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq" event={"ID":"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65","Type":"ContainerDied","Data":"52fe236418e365c85ef771811dd2f05357c2473f31991a49ff8af4432a8c8563"}
Nov 02 01:37:12.171892 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:37:12.171853    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:37:12.499791 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:12.499691    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:12.726258 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:12.726173    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:18.128291 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:18.127468    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k" event={"ID":"99bfc103-2bb9-45ee-a6dc-77f65af421c0","Type":"ContainerStarted","Data":"1de6e6897d6d7926c3ecbb1672b6c651cd0168d23efaa8f11ba2404f42121d7a"}
Nov 02 01:37:19.140073 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:19.140022    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79" event={"ID":"085912b7-b205-4955-be7e-5e4a4b38f179","Type":"ContainerStarted","Data":"db9ccf913e22127af515db57c778e5d2e864b5b8b54a9c3fb4fb2b53b703b20b"}
Nov 02 01:37:19.150796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:19.150618    4912 generic.go:356] "Generic (PLEG): container finished" podID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerID="1de6e6897d6d7926c3ecbb1672b6c651cd0168d23efaa8f11ba2404f42121d7a" exitCode=0
Nov 02 01:37:19.151119 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:19.151093    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k" event={"ID":"99bfc103-2bb9-45ee-a6dc-77f65af421c0","Type":"ContainerDied","Data":"1de6e6897d6d7926c3ecbb1672b6c651cd0168d23efaa8f11ba2404f42121d7a"}
Nov 02 01:37:19.155917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:19.155677    4912 generic.go:356] "Generic (PLEG): container finished" podID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerID="9f89b850a0c2459da8f81a5f631ef3e8f7cf73be21efdf4ab0204247cac0986c" exitCode=0
Nov 02 01:37:19.155917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:19.155766    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl" event={"ID":"be7b7a96-bbac-469f-8320-f1aed5f1f787","Type":"ContainerDied","Data":"9f89b850a0c2459da8f81a5f631ef3e8f7cf73be21efdf4ab0204247cac0986c"}
Nov 02 01:37:19.164886 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:19.164707    4912 generic.go:356] "Generic (PLEG): container finished" podID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerID="79e90198ca3c133c6b0f9bc5f2618d32bf65c6a697b46937790b82f95e352c9c" exitCode=0
Nov 02 01:37:19.164886 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:19.164744    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7" event={"ID":"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3","Type":"ContainerDied","Data":"79e90198ca3c133c6b0f9bc5f2618d32bf65c6a697b46937790b82f95e352c9c"}
Nov 02 01:37:19.262104 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:19.260973    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79" podStartSLOduration=3.29738421 podStartE2EDuration="41.260958724s" totalImagesPullingTime="37.963574514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:19.259960486 +0000 UTC m=+332.165425843" watchObservedRunningTime="2026-11-02 01:37:19.260958724 +0000 UTC m=+332.166424081"
Nov 02 01:37:19.262104 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:19.261446    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k" podStartSLOduration=3.633771381 podStartE2EDuration="41.261440295s" totalImagesPullingTime="37.627668914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:18.253627818 +0000 UTC m=+331.159093177" watchObservedRunningTime="2026-11-02 01:37:19.261440295 +0000 UTC m=+332.166905651"
Nov 02 01:37:20.173375 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.173341    4912 generic.go:356] "Generic (PLEG): container finished" podID="085912b7-b205-4955-be7e-5e4a4b38f179" containerID="db9ccf913e22127af515db57c778e5d2e864b5b8b54a9c3fb4fb2b53b703b20b" exitCode=0
Nov 02 01:37:20.173704 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.173477    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79" event={"ID":"085912b7-b205-4955-be7e-5e4a4b38f179","Type":"ContainerDied","Data":"db9ccf913e22127af515db57c778e5d2e864b5b8b54a9c3fb4fb2b53b703b20b"}
Nov 02 01:37:20.177260 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.177232    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh" event={"ID":"88594ff7-0c92-407f-af92-881d0aaf03f0","Type":"ContainerStarted","Data":"d6007bec7acfbc384fb45f3e375fdc96dcc007c3cb9ab25bd1462e1bb722edcf"}
Nov 02 01:37:20.182301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.182275    4912 generic.go:356] "Generic (PLEG): container finished" podID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerID="9f4dbf129be3b0c7ee35209c12ebbc1815062da1ea36dfeabfb5f309e3bbc392" exitCode=0
Nov 02 01:37:20.182631 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.182608    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9" event={"ID":"baf2ff2c-64d1-44c9-949f-8f3c26ac673a","Type":"ContainerDied","Data":"9f4dbf129be3b0c7ee35209c12ebbc1815062da1ea36dfeabfb5f309e3bbc392"}
Nov 02 01:37:20.408552 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.408496    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh" podStartSLOduration=3.951678985 podStartE2EDuration="42.40847658s" totalImagesPullingTime="38.456797595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:20.406285172 +0000 UTC m=+333.311750555" watchObservedRunningTime="2026-11-02 01:37:20.40847658 +0000 UTC m=+333.313941939"
Nov 02 01:37:20.476043 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.475874    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:37:20.503045 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.502604    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:37:20.523874 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.523848    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:37:20.587515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.587405    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-bundle\") pod \"be7b7a96-bbac-469f-8320-f1aed5f1f787\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") "
Nov 02 01:37:20.587515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.587456    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bfc103-2bb9-45ee-a6dc-77f65af421c0-kube-api-access-jq542\" (UniqueName: \"kubernetes.io/projected/99bfc103-2bb9-45ee-a6dc-77f65af421c0-kube-api-access-jq542\") pod \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") "
Nov 02 01:37:20.587515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.587487    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-util\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-util\") pod \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") "
Nov 02 01:37:20.587712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.587546    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-util\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-util\") pod \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") "
Nov 02 01:37:20.587712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.587576    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-bundle\") pod \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\" (UID: \"99bfc103-2bb9-45ee-a6dc-77f65af421c0\") "
Nov 02 01:37:20.587712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.587629    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-bundle\") pod \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") "
Nov 02 01:37:20.587874 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.587713    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-kube-api-access-2k8b2\" (UniqueName: \"kubernetes.io/projected/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-kube-api-access-2k8b2\") pod \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\" (UID: \"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3\") "
Nov 02 01:37:20.587874 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.587755    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7b7a96-bbac-469f-8320-f1aed5f1f787-kube-api-access-zp6v5\" (UniqueName: \"kubernetes.io/projected/be7b7a96-bbac-469f-8320-f1aed5f1f787-kube-api-access-zp6v5\") pod \"be7b7a96-bbac-469f-8320-f1aed5f1f787\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") "
Nov 02 01:37:20.587874 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.587795    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-util\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-util\") pod \"be7b7a96-bbac-469f-8320-f1aed5f1f787\" (UID: \"be7b7a96-bbac-469f-8320-f1aed5f1f787\") "
Nov 02 01:37:20.588327 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.588284    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-bundle" pod "be7b7a96-bbac-469f-8320-f1aed5f1f787" (UID: "be7b7a96-bbac-469f-8320-f1aed5f1f787"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:20.589019 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.588981    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-bundle" pod "99bfc103-2bb9-45ee-a6dc-77f65af421c0" (UID: "99bfc103-2bb9-45ee-a6dc-77f65af421c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:20.593757 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.593733    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-util" pod "99bfc103-2bb9-45ee-a6dc-77f65af421c0" (UID: "99bfc103-2bb9-45ee-a6dc-77f65af421c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:20.595167 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.595138    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-bundle" pod "4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" (UID: "4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:20.595882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.595858    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-util" pod "be7b7a96-bbac-469f-8320-f1aed5f1f787" (UID: "be7b7a96-bbac-469f-8320-f1aed5f1f787"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:20.596247 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.596227    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-util" pod "4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" (UID: "4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:20.596568 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.596542    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-kube-api-access-2k8b2" pod "4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" (UID: "4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3"). InnerVolumeSpecName "kube-api-access-2k8b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:20.596648 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.596586    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bfc103-2bb9-45ee-a6dc-77f65af421c0-kube-api-access-jq542" pod "99bfc103-2bb9-45ee-a6dc-77f65af421c0" (UID: "99bfc103-2bb9-45ee-a6dc-77f65af421c0"). InnerVolumeSpecName "kube-api-access-jq542". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:20.597955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.597922    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7b7a96-bbac-469f-8320-f1aed5f1f787-kube-api-access-zp6v5" pod "be7b7a96-bbac-469f-8320-f1aed5f1f787" (UID: "be7b7a96-bbac-469f-8320-f1aed5f1f787"). InnerVolumeSpecName "kube-api-access-zp6v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:20.688599 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.688569    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq542\" (UniqueName: \"kubernetes.io/projected/99bfc103-2bb9-45ee-a6dc-77f65af421c0-kube-api-access-jq542\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:20.688712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.688605    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:20.688712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.688622    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:20.688712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.688635    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99bfc103-2bb9-45ee-a6dc-77f65af421c0-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:20.688712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.688651    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:20.688712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.688663    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k8b2\" (UniqueName: \"kubernetes.io/projected/4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3-kube-api-access-2k8b2\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:20.688712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.688677    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6v5\" (UniqueName: \"kubernetes.io/projected/be7b7a96-bbac-469f-8320-f1aed5f1f787-kube-api-access-zp6v5\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:20.688712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.688691    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:20.688712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:20.688705    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7b7a96-bbac-469f-8320-f1aed5f1f787-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:21.192101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.190176    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl" event={"ID":"be7b7a96-bbac-469f-8320-f1aed5f1f787","Type":"ContainerDied","Data":"de86dfa7b5e56d3a10f13d0d9c7e9f62d0e32fe8284ed65bb9b1a21553313ef6"}
Nov 02 01:37:21.192101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.190209    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rpdfl"
Nov 02 01:37:21.192101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.190269    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de86dfa7b5e56d3a10f13d0d9c7e9f62d0e32fe8284ed65bb9b1a21553313ef6"
Nov 02 01:37:21.192967 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.192947    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7"
Nov 02 01:37:21.192967 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.192950    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckk7b7" event={"ID":"4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3","Type":"ContainerDied","Data":"cb136924a3480f5f1958421213d2cba2da52b5be6f785cf0d8462cb25edb98fb"}
Nov 02 01:37:21.193182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.192991    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb136924a3480f5f1958421213d2cba2da52b5be6f785cf0d8462cb25edb98fb"
Nov 02 01:37:21.195461 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.195435    4912 generic.go:356] "Generic (PLEG): container finished" podID="7000ca06-b94f-460d-a298-c7fe79254edd" containerID="0ae0f89fcdc5f0b2a779a9bd3e0b957ad3c3bbd76d7aa6d50a5806da2071c3e8" exitCode=0
Nov 02 01:37:21.195599 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.195574    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97" event={"ID":"7000ca06-b94f-460d-a298-c7fe79254edd","Type":"ContainerDied","Data":"0ae0f89fcdc5f0b2a779a9bd3e0b957ad3c3bbd76d7aa6d50a5806da2071c3e8"}
Nov 02 01:37:21.198215 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.198189    4912 generic.go:356] "Generic (PLEG): container finished" podID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerID="63f7a5aaf7d67e3382ffca0a151782a1216b62bd80e2aa630e7d1d5139e530a0" exitCode=0
Nov 02 01:37:21.198334 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.198271    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z" event={"ID":"14b25417-19b5-4ad0-b6e3-5ec163320330","Type":"ContainerDied","Data":"63f7a5aaf7d67e3382ffca0a151782a1216b62bd80e2aa630e7d1d5139e530a0"}
Nov 02 01:37:21.200827 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.200799    4912 generic.go:356] "Generic (PLEG): container finished" podID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerID="4d9ac7ad3c484ad697f4a2d2b6e5b32e8b8d78cec1c6a2cec7e60b15c316a32f" exitCode=0
Nov 02 01:37:21.200938 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.200876    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st" event={"ID":"7a81a42e-8f2b-4377-9a2d-ad4536e2c436","Type":"ContainerDied","Data":"4d9ac7ad3c484ad697f4a2d2b6e5b32e8b8d78cec1c6a2cec7e60b15c316a32f"}
Nov 02 01:37:21.204170 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.204152    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k"
Nov 02 01:37:21.204312 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.204151    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ncr7k" event={"ID":"99bfc103-2bb9-45ee-a6dc-77f65af421c0","Type":"ContainerDied","Data":"afc42b6221932730155361e0e45858c1118fd331a4042997a48509cd21a209bd"}
Nov 02 01:37:21.204481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.204324    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc42b6221932730155361e0e45858c1118fd331a4042997a48509cd21a209bd"
Nov 02 01:37:21.207107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.207081    4912 generic.go:356] "Generic (PLEG): container finished" podID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerID="d6007bec7acfbc384fb45f3e375fdc96dcc007c3cb9ab25bd1462e1bb722edcf" exitCode=0
Nov 02 01:37:21.207224 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.207112    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh" event={"ID":"88594ff7-0c92-407f-af92-881d0aaf03f0","Type":"ContainerDied","Data":"d6007bec7acfbc384fb45f3e375fdc96dcc007c3cb9ab25bd1462e1bb722edcf"}
Nov 02 01:37:21.430019 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.429994    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:37:21.444052 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.444017    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:37:21.495974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.495941    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-bundle\") pod \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") "
Nov 02 01:37:21.496151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.495989    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-util\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-util\") pod \"085912b7-b205-4955-be7e-5e4a4b38f179\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") "
Nov 02 01:37:21.496151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.496031    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-kube-api-access-47sf2\" (UniqueName: \"kubernetes.io/projected/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-kube-api-access-47sf2\") pod \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") "
Nov 02 01:37:21.496151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.496098    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-bundle\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-bundle\") pod \"085912b7-b205-4955-be7e-5e4a4b38f179\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") "
Nov 02 01:37:21.496341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.496154    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-util\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-util\") pod \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\" (UID: \"baf2ff2c-64d1-44c9-949f-8f3c26ac673a\") "
Nov 02 01:37:21.496341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.496217    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085912b7-b205-4955-be7e-5e4a4b38f179-kube-api-access-mfr2r\" (UniqueName: \"kubernetes.io/projected/085912b7-b205-4955-be7e-5e4a4b38f179-kube-api-access-mfr2r\") pod \"085912b7-b205-4955-be7e-5e4a4b38f179\" (UID: \"085912b7-b205-4955-be7e-5e4a4b38f179\") "
Nov 02 01:37:21.496836 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.496787    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-bundle" pod "085912b7-b205-4955-be7e-5e4a4b38f179" (UID: "085912b7-b205-4955-be7e-5e4a4b38f179"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:21.496953 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.496916    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-bundle" pod "baf2ff2c-64d1-44c9-949f-8f3c26ac673a" (UID: "baf2ff2c-64d1-44c9-949f-8f3c26ac673a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:21.499408 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.499088    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085912b7-b205-4955-be7e-5e4a4b38f179-kube-api-access-mfr2r" pod "085912b7-b205-4955-be7e-5e4a4b38f179" (UID: "085912b7-b205-4955-be7e-5e4a4b38f179"). InnerVolumeSpecName "kube-api-access-mfr2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:21.499521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.499493    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-kube-api-access-47sf2" pod "baf2ff2c-64d1-44c9-949f-8f3c26ac673a" (UID: "baf2ff2c-64d1-44c9-949f-8f3c26ac673a"). InnerVolumeSpecName "kube-api-access-47sf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:21.503348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.503314    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-util" pod "baf2ff2c-64d1-44c9-949f-8f3c26ac673a" (UID: "baf2ff2c-64d1-44c9-949f-8f3c26ac673a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:21.503348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.503327    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-util" pod "085912b7-b205-4955-be7e-5e4a4b38f179" (UID: "085912b7-b205-4955-be7e-5e4a4b38f179"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:21.596905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.596825    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47sf2\" (UniqueName: \"kubernetes.io/projected/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-kube-api-access-47sf2\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:21.596905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.596859    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:21.596905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.596872    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:21.596905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.596884    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfr2r\" (UniqueName: \"kubernetes.io/projected/085912b7-b205-4955-be7e-5e4a4b38f179-kube-api-access-mfr2r\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:21.596905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.596897    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf2ff2c-64d1-44c9-949f-8f3c26ac673a-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:21.596905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:21.596909    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085912b7-b205-4955-be7e-5e4a4b38f179-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.213916 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.213883    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79"
Nov 02 01:37:22.214380 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.213883    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2dw79" event={"ID":"085912b7-b205-4955-be7e-5e4a4b38f179","Type":"ContainerDied","Data":"0791683c688f420fee5342fb8bea6fa6fd91de82891c223e9b6f03425699207c"}
Nov 02 01:37:22.214380 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.214006    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0791683c688f420fee5342fb8bea6fa6fd91de82891c223e9b6f03425699207c"
Nov 02 01:37:22.216296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.216267    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9" event={"ID":"fef8b8f9-2024-4b46-95f0-7f711910f3d1","Type":"ContainerStarted","Data":"28d2b1beb8a9093b719aa17324ba6b9d4410edf5b5987f2b55a5687b97c64d4c"}
Nov 02 01:37:22.218286 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.218259    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9"
Nov 02 01:37:22.218389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.218290    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqs8q9" event={"ID":"baf2ff2c-64d1-44c9-949f-8f3c26ac673a","Type":"ContainerDied","Data":"c82cdd6770a8be0f455f377e25171e9629a86bbbd951da1d215bbe0098153d9a"}
Nov 02 01:37:22.218389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.218328    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c82cdd6770a8be0f455f377e25171e9629a86bbbd951da1d215bbe0098153d9a"
Nov 02 01:37:22.220621 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.220573    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq" event={"ID":"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65","Type":"ContainerStarted","Data":"7817d9bcf976382f27e827871f3bafc8110ec2d47c67ee1cfb5516f38c3eaa59"}
Nov 02 01:37:22.335309 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.335258    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9" podStartSLOduration=4.588629933 podStartE2EDuration="52.335242638s" totalImagesPullingTime="47.746612705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:22.333656982 +0000 UTC m=+335.239122340" watchObservedRunningTime="2026-11-02 01:37:22.335242638 +0000 UTC m=+335.240707994"
Nov 02 01:37:22.430540 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.430511    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:37:22.433625 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.433600    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:37:22.437039 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.437014    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:37:22.440498 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.440479    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:37:22.505094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.504988    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7000ca06-b94f-460d-a298-c7fe79254edd-kube-api-access-ll5dk\" (UniqueName: \"kubernetes.io/projected/7000ca06-b94f-460d-a298-c7fe79254edd-kube-api-access-ll5dk\") pod \"7000ca06-b94f-460d-a298-c7fe79254edd\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") "
Nov 02 01:37:22.505094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505039    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-bundle\") pod \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") "
Nov 02 01:37:22.505094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505092    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-util\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-util\") pod \"7000ca06-b94f-460d-a298-c7fe79254edd\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") "
Nov 02 01:37:22.505349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505082    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq" podStartSLOduration=3.524450133 podStartE2EDuration="50.50506976s" totalImagesPullingTime="46.980619627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:22.50370801 +0000 UTC m=+335.409173368" watchObservedRunningTime="2026-11-02 01:37:22.50506976 +0000 UTC m=+335.410535107"
Nov 02 01:37:22.505349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505134    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88594ff7-0c92-407f-af92-881d0aaf03f0-kube-api-access-6rw6t\" (UniqueName: \"kubernetes.io/projected/88594ff7-0c92-407f-af92-881d0aaf03f0-kube-api-access-6rw6t\") pod \"88594ff7-0c92-407f-af92-881d0aaf03f0\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") "
Nov 02 01:37:22.505349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505186    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-util\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-util\") pod \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") "
Nov 02 01:37:22.505349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505213    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-bundle\") pod \"88594ff7-0c92-407f-af92-881d0aaf03f0\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") "
Nov 02 01:37:22.505349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505250    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-util\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-util\") pod \"88594ff7-0c92-407f-af92-881d0aaf03f0\" (UID: \"88594ff7-0c92-407f-af92-881d0aaf03f0\") "
Nov 02 01:37:22.505349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505280    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-bundle\") pod \"14b25417-19b5-4ad0-b6e3-5ec163320330\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") "
Nov 02 01:37:22.505349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505308    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-kube-api-access-gnf5c\" (UniqueName: \"kubernetes.io/projected/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-kube-api-access-gnf5c\") pod \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\" (UID: \"7a81a42e-8f2b-4377-9a2d-ad4536e2c436\") "
Nov 02 01:37:22.505681 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505360    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-util\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-util\") pod \"14b25417-19b5-4ad0-b6e3-5ec163320330\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") "
Nov 02 01:37:22.505681 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505390    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-bundle\") pod \"7000ca06-b94f-460d-a298-c7fe79254edd\" (UID: \"7000ca06-b94f-460d-a298-c7fe79254edd\") "
Nov 02 01:37:22.505681 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505430    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b25417-19b5-4ad0-b6e3-5ec163320330-kube-api-access-sg8h5\" (UniqueName: \"kubernetes.io/projected/14b25417-19b5-4ad0-b6e3-5ec163320330-kube-api-access-sg8h5\") pod \"14b25417-19b5-4ad0-b6e3-5ec163320330\" (UID: \"14b25417-19b5-4ad0-b6e3-5ec163320330\") "
Nov 02 01:37:22.505963 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.505930    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-bundle" pod "14b25417-19b5-4ad0-b6e3-5ec163320330" (UID: "14b25417-19b5-4ad0-b6e3-5ec163320330"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:22.506239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.506208    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-bundle" pod "7a81a42e-8f2b-4377-9a2d-ad4536e2c436" (UID: "7a81a42e-8f2b-4377-9a2d-ad4536e2c436"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:22.506561 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.506512    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-bundle" pod "88594ff7-0c92-407f-af92-881d0aaf03f0" (UID: "88594ff7-0c92-407f-af92-881d0aaf03f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:22.506630 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.506590    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-bundle" pod "7000ca06-b94f-460d-a298-c7fe79254edd" (UID: "7000ca06-b94f-460d-a298-c7fe79254edd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:22.508759 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.508733    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7000ca06-b94f-460d-a298-c7fe79254edd-kube-api-access-ll5dk" pod "7000ca06-b94f-460d-a298-c7fe79254edd" (UID: "7000ca06-b94f-460d-a298-c7fe79254edd"). InnerVolumeSpecName "kube-api-access-ll5dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:22.508860 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.508760    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b25417-19b5-4ad0-b6e3-5ec163320330-kube-api-access-sg8h5" pod "14b25417-19b5-4ad0-b6e3-5ec163320330" (UID: "14b25417-19b5-4ad0-b6e3-5ec163320330"). InnerVolumeSpecName "kube-api-access-sg8h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:22.508914 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.508892    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-kube-api-access-gnf5c" pod "7a81a42e-8f2b-4377-9a2d-ad4536e2c436" (UID: "7a81a42e-8f2b-4377-9a2d-ad4536e2c436"). InnerVolumeSpecName "kube-api-access-gnf5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:22.509127 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.509106    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88594ff7-0c92-407f-af92-881d0aaf03f0-kube-api-access-6rw6t" pod "88594ff7-0c92-407f-af92-881d0aaf03f0" (UID: "88594ff7-0c92-407f-af92-881d0aaf03f0"). InnerVolumeSpecName "kube-api-access-6rw6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:22.513797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.513776    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-util" pod "7a81a42e-8f2b-4377-9a2d-ad4536e2c436" (UID: "7a81a42e-8f2b-4377-9a2d-ad4536e2c436"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:22.514203 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.514187    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-util" pod "7000ca06-b94f-460d-a298-c7fe79254edd" (UID: "7000ca06-b94f-460d-a298-c7fe79254edd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:22.514247 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.514203    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-util" pod "88594ff7-0c92-407f-af92-881d0aaf03f0" (UID: "88594ff7-0c92-407f-af92-881d0aaf03f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:22.514509 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.514490    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-util" pod "14b25417-19b5-4ad0-b6e3-5ec163320330" (UID: "14b25417-19b5-4ad0-b6e3-5ec163320330"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:22.607081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607013    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607053    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607082    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg8h5\" (UniqueName: \"kubernetes.io/projected/14b25417-19b5-4ad0-b6e3-5ec163320330-kube-api-access-sg8h5\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607092    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll5dk\" (UniqueName: \"kubernetes.io/projected/7000ca06-b94f-460d-a298-c7fe79254edd-kube-api-access-ll5dk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607126    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607135    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7000ca06-b94f-460d-a298-c7fe79254edd-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607143    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rw6t\" (UniqueName: \"kubernetes.io/projected/88594ff7-0c92-407f-af92-881d0aaf03f0-kube-api-access-6rw6t\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607152    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607160    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607167    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88594ff7-0c92-407f-af92-881d0aaf03f0-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607175    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14b25417-19b5-4ad0-b6e3-5ec163320330-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:22.607296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:22.607184    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnf5c\" (UniqueName: \"kubernetes.io/projected/7a81a42e-8f2b-4377-9a2d-ad4536e2c436-kube-api-access-gnf5c\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:23.227865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.227778    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97" event={"ID":"7000ca06-b94f-460d-a298-c7fe79254edd","Type":"ContainerDied","Data":"c290c4d748c4e3562d25ad1aa6625f40306ed77129a87a240c4cbb78c585be45"}
Nov 02 01:37:23.227865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.227800    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x5j97"
Nov 02 01:37:23.227865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.227840    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c290c4d748c4e3562d25ad1aa6625f40306ed77129a87a240c4cbb78c585be45"
Nov 02 01:37:23.230404 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.230375    4912 generic.go:356] "Generic (PLEG): container finished" podID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerID="7817d9bcf976382f27e827871f3bafc8110ec2d47c67ee1cfb5516f38c3eaa59" exitCode=0
Nov 02 01:37:23.230883 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.230447    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq" event={"ID":"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65","Type":"ContainerDied","Data":"7817d9bcf976382f27e827871f3bafc8110ec2d47c67ee1cfb5516f38c3eaa59"}
Nov 02 01:37:23.232822 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.232805    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z"
Nov 02 01:37:23.232930 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.232835    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lfx2z" event={"ID":"14b25417-19b5-4ad0-b6e3-5ec163320330","Type":"ContainerDied","Data":"1b296e346f61b823e8896b44998cfe365c63f42b0864dfdfadd352898536a1e6"}
Nov 02 01:37:23.232930 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.232874    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b296e346f61b823e8896b44998cfe365c63f42b0864dfdfadd352898536a1e6"
Nov 02 01:37:23.235469 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.235445    4912 generic.go:356] "Generic (PLEG): container finished" podID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerID="28d2b1beb8a9093b719aa17324ba6b9d4410edf5b5987f2b55a5687b97c64d4c" exitCode=0
Nov 02 01:37:23.235560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.235487    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9" event={"ID":"fef8b8f9-2024-4b46-95f0-7f711910f3d1","Type":"ContainerDied","Data":"28d2b1beb8a9093b719aa17324ba6b9d4410edf5b5987f2b55a5687b97c64d4c"}
Nov 02 01:37:23.237683 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.237665    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st"
Nov 02 01:37:23.237793 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.237693    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pn7st" event={"ID":"7a81a42e-8f2b-4377-9a2d-ad4536e2c436","Type":"ContainerDied","Data":"088ec041a42dc372d84a3526afbd387608f34a68f5e43c03575a2cc867275ce6"}
Nov 02 01:37:23.237793 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.237728    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="088ec041a42dc372d84a3526afbd387608f34a68f5e43c03575a2cc867275ce6"
Nov 02 01:37:23.239914 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.239892    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh" event={"ID":"88594ff7-0c92-407f-af92-881d0aaf03f0","Type":"ContainerDied","Data":"da49da6bbc14cd4f88f45e754e1002da36ec76ff4e005568a6431d909ef63efa"}
Nov 02 01:37:23.240020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.239936    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m7hsh"
Nov 02 01:37:23.240020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:23.239964    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da49da6bbc14cd4f88f45e754e1002da36ec76ff4e005568a6431d909ef63efa"
Nov 02 01:37:24.381859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.381831    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:37:24.408705 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.408685    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:37:24.522119 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.522014    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-util\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-util\") pod \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") "
Nov 02 01:37:24.522119 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.522103    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-bundle\") pod \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") "
Nov 02 01:37:24.522348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.522148    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-kube-api-access-bgxvk\" (UniqueName: \"kubernetes.io/projected/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-kube-api-access-bgxvk\") pod \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") "
Nov 02 01:37:24.522348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.522234    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef8b8f9-2024-4b46-95f0-7f711910f3d1-kube-api-access-zzbtn\" (UniqueName: \"kubernetes.io/projected/fef8b8f9-2024-4b46-95f0-7f711910f3d1-kube-api-access-zzbtn\") pod \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") "
Nov 02 01:37:24.522348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.522280    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-bundle\") pod \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\" (UID: \"fef8b8f9-2024-4b46-95f0-7f711910f3d1\") "
Nov 02 01:37:24.522546 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.522350    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-util\") pod \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\" (UID: \"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65\") "
Nov 02 01:37:24.523168 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.523145    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-bundle" pod "fef8b8f9-2024-4b46-95f0-7f711910f3d1" (UID: "fef8b8f9-2024-4b46-95f0-7f711910f3d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:24.523455 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.523428    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-bundle" pod "e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" (UID: "e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:24.524783 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.524758    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-kube-api-access-bgxvk" pod "e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" (UID: "e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65"). InnerVolumeSpecName "kube-api-access-bgxvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:24.525105 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.525079    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef8b8f9-2024-4b46-95f0-7f711910f3d1-kube-api-access-zzbtn" pod "fef8b8f9-2024-4b46-95f0-7f711910f3d1" (UID: "fef8b8f9-2024-4b46-95f0-7f711910f3d1"). InnerVolumeSpecName "kube-api-access-zzbtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:37:24.527407 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.527388    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-util" pod "fef8b8f9-2024-4b46-95f0-7f711910f3d1" (UID: "fef8b8f9-2024-4b46-95f0-7f711910f3d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:24.531137 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.531113    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-util" pod "e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" (UID: "e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:37:24.623986 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.623962    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:24.623986 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.623984    4912 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-util\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:24.623986 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.623992    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:24.624224 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.624000    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgxvk\" (UniqueName: \"kubernetes.io/projected/e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65-kube-api-access-bgxvk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:24.624224 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.624010    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzbtn\" (UniqueName: \"kubernetes.io/projected/fef8b8f9-2024-4b46-95f0-7f711910f3d1-kube-api-access-zzbtn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:24.624224 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:24.624032    4912 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef8b8f9-2024-4b46-95f0-7f711910f3d1-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:37:25.250684 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:25.250658    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9"
Nov 02 01:37:25.250828 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:25.250706    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cggcc9" event={"ID":"fef8b8f9-2024-4b46-95f0-7f711910f3d1","Type":"ContainerDied","Data":"a523c5a0d6bdcd24a2b1d6e6565950a8e17c05e12768506125e9ed519c8df677"}
Nov 02 01:37:25.250828 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:25.250757    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a523c5a0d6bdcd24a2b1d6e6565950a8e17c05e12768506125e9ed519c8df677"
Nov 02 01:37:25.252923 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:25.252899    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq" event={"ID":"e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65","Type":"ContainerDied","Data":"33cc8fff3502507a243f3443b22b27b6a8fb48b454b88ca88e6a2b06a8bc4364"}
Nov 02 01:37:25.253030 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:25.252931    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33cc8fff3502507a243f3443b22b27b6a8fb48b454b88ca88e6a2b06a8bc4364"
Nov 02 01:37:25.253030 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:25.252933    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqpnpq"
Nov 02 01:37:26.687638 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:26.687603    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:27.489916 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:27.489883    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:28.491642 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:28.491610    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:32.299547 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.299504    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:32.543522 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.543487    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:32.690379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690350    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"]
Nov 02 01:37:32.690662 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690648    4912 cpu_manager.go:414] "containerMap: removing container" podUID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerName="extract"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690664    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerName="extract"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690673    4912 cpu_manager.go:414] "containerMap: removing container" podUID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerName="extract"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690678    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerName="extract"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690688    4912 cpu_manager.go:414] "containerMap: removing container" podUID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerName="util"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690692    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerName="util"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690699    4912 cpu_manager.go:414] "containerMap: removing container" podUID="085912b7-b205-4955-be7e-5e4a4b38f179" containerName="util"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690702    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085912b7-b205-4955-be7e-5e4a4b38f179" containerName="util"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690710    4912 cpu_manager.go:414] "containerMap: removing container" podUID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerName="pull"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690715    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerName="pull"
Nov 02 01:37:32.690719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690720    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690724    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690732    4912 cpu_manager.go:414] "containerMap: removing container" podUID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690736    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690742    4912 cpu_manager.go:414] "containerMap: removing container" podUID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690746    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690752    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7000ca06-b94f-460d-a298-c7fe79254edd" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690755    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7000ca06-b94f-460d-a298-c7fe79254edd" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690761    4912 cpu_manager.go:414] "containerMap: removing container" podUID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690765    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690770    4912 cpu_manager.go:414] "containerMap: removing container" podUID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690773    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690779    4912 cpu_manager.go:414] "containerMap: removing container" podUID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690782    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690787    4912 cpu_manager.go:414] "containerMap: removing container" podUID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690791    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690801    4912 cpu_manager.go:414] "containerMap: removing container" podUID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690806    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690813    4912 cpu_manager.go:414] "containerMap: removing container" podUID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690817    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690825    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7000ca06-b94f-460d-a298-c7fe79254edd" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690829    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7000ca06-b94f-460d-a298-c7fe79254edd" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690836    4912 cpu_manager.go:414] "containerMap: removing container" podUID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690840    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690845    4912 cpu_manager.go:414] "containerMap: removing container" podUID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690850    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690855    4912 cpu_manager.go:414] "containerMap: removing container" podUID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690859    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690864    4912 cpu_manager.go:414] "containerMap: removing container" podUID="085912b7-b205-4955-be7e-5e4a4b38f179" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690868    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085912b7-b205-4955-be7e-5e4a4b38f179" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690875    4912 cpu_manager.go:414] "containerMap: removing container" podUID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690879    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690886    4912 cpu_manager.go:414] "containerMap: removing container" podUID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690889    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690894    4912 cpu_manager.go:414] "containerMap: removing container" podUID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690898    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690903    4912 cpu_manager.go:414] "containerMap: removing container" podUID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690907    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690912    4912 cpu_manager.go:414] "containerMap: removing container" podUID="085912b7-b205-4955-be7e-5e4a4b38f179" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690916    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085912b7-b205-4955-be7e-5e4a4b38f179" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690927    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690931    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690937    4912 cpu_manager.go:414] "containerMap: removing container" podUID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690940    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690946    4912 cpu_manager.go:414] "containerMap: removing container" podUID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690950    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690954    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690958    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerName="util"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690964    4912 cpu_manager.go:414] "containerMap: removing container" podUID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690969    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerName="pull"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690975    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7000ca06-b94f-460d-a298-c7fe79254edd" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690979    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7000ca06-b94f-460d-a298-c7fe79254edd" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690988    4912 cpu_manager.go:414] "containerMap: removing container" podUID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690992    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.690997    4912 cpu_manager.go:414] "containerMap: removing container" podUID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691002    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691047    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bfc103-2bb9-45ee-a6dc-77f65af421c0" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691072    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="88594ff7-0c92-407f-af92-881d0aaf03f0" containerName="extract"
Nov 02 01:37:32.691023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691078    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a81a42e-8f2b-4377-9a2d-ad4536e2c436" containerName="extract"
Nov 02 01:37:32.692753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691084    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c6a022-03df-4bb4-9d2b-8adeaa3d8d65" containerName="extract"
Nov 02 01:37:32.692753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691089    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf2ff2c-64d1-44c9-949f-8f3c26ac673a" containerName="extract"
Nov 02 01:37:32.692753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691094    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd9d7ee-4fe3-4dcf-8e82-baea45d185c3" containerName="extract"
Nov 02 01:37:32.692753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691098    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef8b8f9-2024-4b46-95f0-7f711910f3d1" containerName="extract"
Nov 02 01:37:32.692753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691105    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="085912b7-b205-4955-be7e-5e4a4b38f179" containerName="extract"
Nov 02 01:37:32.692753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691111    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="7000ca06-b94f-460d-a298-c7fe79254edd" containerName="extract"
Nov 02 01:37:32.692753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691116    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7b7a96-bbac-469f-8320-f1aed5f1f787" containerName="extract"
Nov 02 01:37:32.692753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.691121    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b25417-19b5-4ad0-b6e3-5ec163320330" containerName="extract"
Nov 02 01:37:32.693095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.693079    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:32.748234 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.748205    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"]
Nov 02 01:37:32.789726 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.789691    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70180f03-499e-439d-b3ba-ea7482b78cf2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cfd42\" (UID: \"70180f03-499e-439d-b3ba-ea7482b78cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:32.789896 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.789769    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kdhd\" (UniqueName: \"kubernetes.io/projected/70180f03-499e-439d-b3ba-ea7482b78cf2-kube-api-access-7kdhd\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cfd42\" (UID: \"70180f03-499e-439d-b3ba-ea7482b78cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:32.795552 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.795524    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 01:37:32.795688 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.795635    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fbj5j\""
Nov 02 01:37:32.795688 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.795635    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 01:37:32.890750 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.890714    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdhd\" (UniqueName: \"kubernetes.io/projected/70180f03-499e-439d-b3ba-ea7482b78cf2-kube-api-access-7kdhd\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cfd42\" (UID: \"70180f03-499e-439d-b3ba-ea7482b78cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:32.890920 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.890765    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70180f03-499e-439d-b3ba-ea7482b78cf2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cfd42\" (UID: \"70180f03-499e-439d-b3ba-ea7482b78cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:32.891192 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.891173    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70180f03-499e-439d-b3ba-ea7482b78cf2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cfd42\" (UID: \"70180f03-499e-439d-b3ba-ea7482b78cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:32.954320 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:32.954263    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdhd\" (UniqueName: \"kubernetes.io/projected/70180f03-499e-439d-b3ba-ea7482b78cf2-kube-api-access-7kdhd\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cfd42\" (UID: \"70180f03-499e-439d-b3ba-ea7482b78cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:33.003857 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:33.003816    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:33.494644 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:33.494605    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"]
Nov 02 01:37:33.532910 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:37:33.532880    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70180f03_499e_439d_b3ba_ea7482b78cf2.slice/crio-60bbc7d8879331cc387071b2aa238bc6c7d5c4f1aa677ba4935dbce2c00db845 WatchSource:0}: Error finding container 60bbc7d8879331cc387071b2aa238bc6c7d5c4f1aa677ba4935dbce2c00db845: Status 404 returned error can't find the container with id 60bbc7d8879331cc387071b2aa238bc6c7d5c4f1aa677ba4935dbce2c00db845
Nov 02 01:37:34.295801 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:34.295752    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42" event={"ID":"70180f03-499e-439d-b3ba-ea7482b78cf2","Type":"ContainerStarted","Data":"60bbc7d8879331cc387071b2aa238bc6c7d5c4f1aa677ba4935dbce2c00db845"}
Nov 02 01:37:34.305560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:34.305530    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:36.538962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:36.538914    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"]
Nov 02 01:37:36.541178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:36.541159    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"
Nov 02 01:37:36.624707 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:36.624671    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbb5\" (UniqueName: \"kubernetes.io/projected/2a3f2067-4ebd-4f7b-92c6-89715e73ac90-kube-api-access-hsbb5\") pod \"limitador-operator-controller-manager-5778bb4449-pvcfb\" (UID: \"2a3f2067-4ebd-4f7b-92c6-89715e73ac90\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"
Nov 02 01:37:36.642618 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:36.642588    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-n2m6z\""
Nov 02 01:37:36.645141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:36.645114    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"]
Nov 02 01:37:36.725479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:36.725426    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbb5\" (UniqueName: \"kubernetes.io/projected/2a3f2067-4ebd-4f7b-92c6-89715e73ac90-kube-api-access-hsbb5\") pod \"limitador-operator-controller-manager-5778bb4449-pvcfb\" (UID: \"2a3f2067-4ebd-4f7b-92c6-89715e73ac90\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"
Nov 02 01:37:36.792569 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:36.792497    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbb5\" (UniqueName: \"kubernetes.io/projected/2a3f2067-4ebd-4f7b-92c6-89715e73ac90-kube-api-access-hsbb5\") pod \"limitador-operator-controller-manager-5778bb4449-pvcfb\" (UID: \"2a3f2067-4ebd-4f7b-92c6-89715e73ac90\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"
Nov 02 01:37:36.856351 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:36.856309    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"
Nov 02 01:37:37.252000 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:37.251927    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"]
Nov 02 01:37:37.254409 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:37:37.254378    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3f2067_4ebd_4f7b_92c6_89715e73ac90.slice/crio-fd22c44f34ade46581ac607d3d28032dca93bda977dacf1694a740080e7faacb WatchSource:0}: Error finding container fd22c44f34ade46581ac607d3d28032dca93bda977dacf1694a740080e7faacb: Status 404 returned error can't find the container with id fd22c44f34ade46581ac607d3d28032dca93bda977dacf1694a740080e7faacb
Nov 02 01:37:37.305817 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:37.305782    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:37.315503 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:37.315469    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb" event={"ID":"2a3f2067-4ebd-4f7b-92c6-89715e73ac90","Type":"ContainerStarted","Data":"fd22c44f34ade46581ac607d3d28032dca93bda977dacf1694a740080e7faacb"}
Nov 02 01:37:38.087887 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:38.087852    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:39.383743 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:39.383699    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:40.332144 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:40.332107    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42" event={"ID":"70180f03-499e-439d-b3ba-ea7482b78cf2","Type":"ContainerStarted","Data":"e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642"}
Nov 02 01:37:40.332327 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:40.332306    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:40.553293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:40.553181    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42" podStartSLOduration=2.760453652 podStartE2EDuration="8.553161453s" totalImagesPullingTime="5.792707801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:40.501349662 +0000 UTC m=+353.406815029" watchObservedRunningTime="2026-11-02 01:37:40.553161453 +0000 UTC m=+353.458626811"
Nov 02 01:37:41.227329 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:41.227293    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:41.343225 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:41.343104    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb" event={"ID":"2a3f2067-4ebd-4f7b-92c6-89715e73ac90","Type":"ContainerStarted","Data":"284f36c8fe4ce7131f7220c5b7e5286d57c94fb4d4e7c2b561b063159cfb5190"}
Nov 02 01:37:41.343455 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:41.343418    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"
Nov 02 01:37:41.459174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:41.459119    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb" podStartSLOduration=1.679696818 podStartE2EDuration="5.459099966s" totalImagesPullingTime="3.779403148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:41.458774589 +0000 UTC m=+354.364239946" watchObservedRunningTime="2026-11-02 01:37:41.459099966 +0000 UTC m=+354.364565325"
Nov 02 01:37:41.895266 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:41.895232    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"]
Nov 02 01:37:41.897331 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:41.897307    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"
Nov 02 01:37:41.897527 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:41.897408    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:41.964544 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:41.964519    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jg9fg\""
Nov 02 01:37:41.969919 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:41.969892    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzs6\" (UniqueName: \"kubernetes.io/projected/0754beef-39a5-42f3-affc-ba64da2c458d-kube-api-access-2pzs6\") pod \"authorino-operator-7b7bb4bc77-2hxrh\" (UID: \"0754beef-39a5-42f3-affc-ba64da2c458d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"
Nov 02 01:37:42.007206 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:42.007175    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"]
Nov 02 01:37:42.071229 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:42.071202    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzs6\" (UniqueName: \"kubernetes.io/projected/0754beef-39a5-42f3-affc-ba64da2c458d-kube-api-access-2pzs6\") pod \"authorino-operator-7b7bb4bc77-2hxrh\" (UID: \"0754beef-39a5-42f3-affc-ba64da2c458d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"
Nov 02 01:37:42.094426 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:42.094394    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:42.132264 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:42.132234    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzs6\" (UniqueName: \"kubernetes.io/projected/0754beef-39a5-42f3-affc-ba64da2c458d-kube-api-access-2pzs6\") pod \"authorino-operator-7b7bb4bc77-2hxrh\" (UID: \"0754beef-39a5-42f3-affc-ba64da2c458d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"
Nov 02 01:37:42.137523 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:37:42.137483    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:37:42.211072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:42.210978    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"
Nov 02 01:37:42.583803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:42.583781    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"]
Nov 02 01:37:42.585820 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:37:42.585792    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0754beef_39a5_42f3_affc_ba64da2c458d.slice/crio-d9509ae86439606d8eddb83457bb487e09f636a8279d84850f5b79456d324252 WatchSource:0}: Error finding container d9509ae86439606d8eddb83457bb487e09f636a8279d84850f5b79456d324252: Status 404 returned error can't find the container with id d9509ae86439606d8eddb83457bb487e09f636a8279d84850f5b79456d324252
Nov 02 01:37:43.262126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.262094    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:43.353274 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.353237    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh" event={"ID":"0754beef-39a5-42f3-affc-ba64da2c458d","Type":"ContainerStarted","Data":"d9509ae86439606d8eddb83457bb487e09f636a8279d84850f5b79456d324252"}
Nov 02 01:37:43.529985 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.529911    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"]
Nov 02 01:37:43.553308 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.553274    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:43.562399 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.562374    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.666886 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.666855    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"]
Nov 02 01:37:43.682125 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.682097    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c067698-ad60-4180-8859-f46e90cfa8d0-metrics-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.682278 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.682146    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6c067698-ad60-4180-8859-f46e90cfa8d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.682278 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.682167    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cc4z\" (UniqueName: \"kubernetes.io/projected/6c067698-ad60-4180-8859-f46e90cfa8d0-kube-api-access-7cc4z\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.682389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.682321    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6c067698-ad60-4180-8859-f46e90cfa8d0-manifests\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.682389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.682373    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c067698-ad60-4180-8859-f46e90cfa8d0-tmp\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.682473 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.682399    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c067698-ad60-4180-8859-f46e90cfa8d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.714425 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.714380    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 01:37:43.714425 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.714418    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 01:37:43.714633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.714433    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 01:37:43.714633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.714447    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-gnd7s\""
Nov 02 01:37:43.714633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.714486    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 01:37:43.714633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.714491    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 01:37:43.733466 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.733438    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:43.783537 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.783466    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6c067698-ad60-4180-8859-f46e90cfa8d0-manifests\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.783537 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.783508    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c067698-ad60-4180-8859-f46e90cfa8d0-tmp\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.783537 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.783534    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c067698-ad60-4180-8859-f46e90cfa8d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.783798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.783551    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c067698-ad60-4180-8859-f46e90cfa8d0-metrics-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.783798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.783584    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6c067698-ad60-4180-8859-f46e90cfa8d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.783798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.783601    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cc4z\" (UniqueName: \"kubernetes.io/projected/6c067698-ad60-4180-8859-f46e90cfa8d0-kube-api-access-7cc4z\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.783956 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.783939    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6c067698-ad60-4180-8859-f46e90cfa8d0-manifests\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.786076 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.786036    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c067698-ad60-4180-8859-f46e90cfa8d0-tmp\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.786333 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.786308    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6c067698-ad60-4180-8859-f46e90cfa8d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.786467 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.786334    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c067698-ad60-4180-8859-f46e90cfa8d0-metrics-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.786510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.786472    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c067698-ad60-4180-8859-f46e90cfa8d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.852858 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.852824    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cc4z\" (UniqueName: \"kubernetes.io/projected/6c067698-ad60-4180-8859-f46e90cfa8d0-kube-api-access-7cc4z\") pod \"opendatahub-operator-controller-manager-6755fff5c-jlqt6\" (UID: \"6c067698-ad60-4180-8859-f46e90cfa8d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.914828 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.914784    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:43.919374 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:43.919340    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:44.099726 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:44.099652    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:44.289459 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:44.289429    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:44.475544 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:37:44.475506    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c067698_ad60_4180_8859_f46e90cfa8d0.slice/crio-2160e4e73bf71b3df3179b3cd5d3bcd3f3166de4a4485f7712a3f013fa84b369 WatchSource:0}: Error finding container 2160e4e73bf71b3df3179b3cd5d3bcd3f3166de4a4485f7712a3f013fa84b369: Status 404 returned error can't find the container with id 2160e4e73bf71b3df3179b3cd5d3bcd3f3166de4a4485f7712a3f013fa84b369
Nov 02 01:37:44.475706 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:44.475670    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"]
Nov 02 01:37:44.527383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:44.527356    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:44.715416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:44.715381    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:44.890767 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:44.890738    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:45.355810 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:45.355630    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:45.366454 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:45.366417    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6" event={"ID":"6c067698-ad60-4180-8859-f46e90cfa8d0","Type":"ContainerStarted","Data":"2160e4e73bf71b3df3179b3cd5d3bcd3f3166de4a4485f7712a3f013fa84b369"}
Nov 02 01:37:45.545740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:45.545637    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:45.738698 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:45.738657    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:46.000483 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.000364    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:46.318264 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.318182    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:46.374965 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.374928    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh" event={"ID":"0754beef-39a5-42f3-affc-ba64da2c458d","Type":"ContainerStarted","Data":"fb5375c77683ed42691e1265ff224dbc0ce1a63c869a30c639b41d14adf0dcc5"}
Nov 02 01:37:46.375790 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.375756    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"
Nov 02 01:37:46.498527 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.498466    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh" podStartSLOduration=2.665768705 podStartE2EDuration="5.498446033s" totalImagesPullingTime="2.832677328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:46.49527945 +0000 UTC m=+359.400744807" watchObservedRunningTime="2026-11-02 01:37:46.498446033 +0000 UTC m=+359.403911391"
Nov 02 01:37:46.796159 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.796126    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"]
Nov 02 01:37:46.800689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.800664    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"
Nov 02 01:37:46.855740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.855704    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 01:37:46.855902 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.855882    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tf6kq\""
Nov 02 01:37:46.907542 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.907506    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"]
Nov 02 01:37:46.913004 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:46.912980    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66b9b\" (UniqueName: \"kubernetes.io/projected/f2037c89-aaf1-459d-9676-593d32c0ce50-kube-api-access-66b9b\") pod \"dns-operator-controller-manager-8557cbdb84-j5pnp\" (UID: \"f2037c89-aaf1-459d-9676-593d32c0ce50\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"
Nov 02 01:37:47.013833 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:47.013790    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66b9b\" (UniqueName: \"kubernetes.io/projected/f2037c89-aaf1-459d-9676-593d32c0ce50-kube-api-access-66b9b\") pod \"dns-operator-controller-manager-8557cbdb84-j5pnp\" (UID: \"f2037c89-aaf1-459d-9676-593d32c0ce50\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"
Nov 02 01:37:47.075100 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:47.075003    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66b9b\" (UniqueName: \"kubernetes.io/projected/f2037c89-aaf1-459d-9676-593d32c0ce50-kube-api-access-66b9b\") pod \"dns-operator-controller-manager-8557cbdb84-j5pnp\" (UID: \"f2037c89-aaf1-459d-9676-593d32c0ce50\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"
Nov 02 01:37:47.115272 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:47.115237    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"
Nov 02 01:37:47.432703 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:47.432665    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:47.497433 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:47.497394    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"]
Nov 02 01:37:47.501431 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:37:47.501402    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2037c89_aaf1_459d_9676_593d32c0ce50.slice/crio-fe3cd06bc5a0066bb7ff96cdcde6fdcc6a8372055813a9fc8ccd0e282a2e5a31 WatchSource:0}: Error finding container fe3cd06bc5a0066bb7ff96cdcde6fdcc6a8372055813a9fc8ccd0e282a2e5a31: Status 404 returned error can't find the container with id fe3cd06bc5a0066bb7ff96cdcde6fdcc6a8372055813a9fc8ccd0e282a2e5a31
Nov 02 01:37:47.666943 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:47.666899    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:48.150697 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:48.150657    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:48.387117 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:48.387081    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp" event={"ID":"f2037c89-aaf1-459d-9676-593d32c0ce50","Type":"ContainerStarted","Data":"fe3cd06bc5a0066bb7ff96cdcde6fdcc6a8372055813a9fc8ccd0e282a2e5a31"}
Nov 02 01:37:48.388785 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:48.388756    4912 generic.go:356] "Generic (PLEG): container finished" podID="6c067698-ad60-4180-8859-f46e90cfa8d0" containerID="95ce7d66501ef4217ce69fcae036be936461fb56434209029321f563b889af39" exitCode=0
Nov 02 01:37:48.388928 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:48.388849    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6" event={"ID":"6c067698-ad60-4180-8859-f46e90cfa8d0","Type":"ContainerDied","Data":"95ce7d66501ef4217ce69fcae036be936461fb56434209029321f563b889af39"}
Nov 02 01:37:48.448881 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:48.448768    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:48.653431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:48.653398    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:49.402406 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:49.402366    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6" event={"ID":"6c067698-ad60-4180-8859-f46e90cfa8d0","Type":"ContainerStarted","Data":"3f2ef127dd4e8336e8aef57d5d6484f810aaa109862eb577d788942d5cb34942"}
Nov 02 01:37:49.402603 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:49.402559    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:37:49.593175 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:49.593075    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6" podStartSLOduration=2.081866787 podStartE2EDuration="6.593045054s" totalImagesPullingTime="4.072023017s" totalInitContainerRuntime="439.15525ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:49.592042796 +0000 UTC m=+362.497508155" watchObservedRunningTime="2026-11-02 01:37:49.593045054 +0000 UTC m=+362.498510411"
Nov 02 01:37:50.225609 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.225580    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"]
Nov 02 01:37:50.229108 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.229087    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"
Nov 02 01:37:50.296034 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.296005    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 01:37:50.296034 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.296034    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:37:50.296293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.296202    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qd2bg\""
Nov 02 01:37:50.319711 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.319682    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"]
Nov 02 01:37:50.344134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.344105    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-922b8\" (UniqueName: \"kubernetes.io/projected/dd4641be-4555-471f-ba3a-ed2b4c7b9b8e-kube-api-access-922b8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq\" (UID: \"dd4641be-4555-471f-ba3a-ed2b4c7b9b8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"
Nov 02 01:37:50.344282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.344149    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd4641be-4555-471f-ba3a-ed2b4c7b9b8e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq\" (UID: \"dd4641be-4555-471f-ba3a-ed2b4c7b9b8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"
Nov 02 01:37:50.445289 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.445257    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-922b8\" (UniqueName: \"kubernetes.io/projected/dd4641be-4555-471f-ba3a-ed2b4c7b9b8e-kube-api-access-922b8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq\" (UID: \"dd4641be-4555-471f-ba3a-ed2b4c7b9b8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"
Nov 02 01:37:50.445461 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.445303    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd4641be-4555-471f-ba3a-ed2b4c7b9b8e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq\" (UID: \"dd4641be-4555-471f-ba3a-ed2b4c7b9b8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"
Nov 02 01:37:50.445638 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.445623    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd4641be-4555-471f-ba3a-ed2b4c7b9b8e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq\" (UID: \"dd4641be-4555-471f-ba3a-ed2b4c7b9b8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"
Nov 02 01:37:50.506425 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.506344    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-922b8\" (UniqueName: \"kubernetes.io/projected/dd4641be-4555-471f-ba3a-ed2b4c7b9b8e-kube-api-access-922b8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq\" (UID: \"dd4641be-4555-471f-ba3a-ed2b4c7b9b8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"
Nov 02 01:37:50.540448 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.540413    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"
Nov 02 01:37:50.974247 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:50.974221    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq"]
Nov 02 01:37:50.994440 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:37:50.994408    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4641be_4555_471f_ba3a_ed2b4c7b9b8e.slice/crio-e83fa43d1617c01fa7afb5748428c4e5ad265ee0fd4176473646356e9a9d40ad WatchSource:0}: Error finding container e83fa43d1617c01fa7afb5748428c4e5ad265ee0fd4176473646356e9a9d40ad: Status 404 returned error can't find the container with id e83fa43d1617c01fa7afb5748428c4e5ad265ee0fd4176473646356e9a9d40ad
Nov 02 01:37:51.056794 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:51.056763    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 01:37:51.345711 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:51.345641    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:37:51.414912 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:51.414879    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp" event={"ID":"f2037c89-aaf1-459d-9676-593d32c0ce50","Type":"ContainerStarted","Data":"ac2e512113d14723e8d19d3116c1f7a1b51e41ff2ab408f3dca4d0260d313a68"}
Nov 02 01:37:51.415105 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:51.414948    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"
Nov 02 01:37:51.416415 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:51.416392    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq" event={"ID":"dd4641be-4555-471f-ba3a-ed2b4c7b9b8e","Type":"ContainerStarted","Data":"e83fa43d1617c01fa7afb5748428c4e5ad265ee0fd4176473646356e9a9d40ad"}
Nov 02 01:37:52.015236 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.015180    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp" podStartSLOduration=2.514486566 podStartE2EDuration="6.015163727s" totalImagesPullingTime="3.500677161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:51.587797057 +0000 UTC m=+364.493262415" watchObservedRunningTime="2026-11-02 01:37:52.015163727 +0000 UTC m=+364.920629101"
Nov 02 01:37:52.017132 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.017110    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"]
Nov 02 01:37:52.020501 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.020486    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:37:52.125917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.125887    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"]
Nov 02 01:37:52.160251 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.160218    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5cb\" (UniqueName: \"kubernetes.io/projected/579e48ba-c588-4231-91ef-61c31ad241c4-kube-api-access-kc5cb\") pod \"kuadrant-operator-controller-manager-678d87766f-cvpvp\" (UID: \"579e48ba-c588-4231-91ef-61c31ad241c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:37:52.160401 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.160351    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/579e48ba-c588-4231-91ef-61c31ad241c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cvpvp\" (UID: \"579e48ba-c588-4231-91ef-61c31ad241c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:37:52.260974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.260939    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/579e48ba-c588-4231-91ef-61c31ad241c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cvpvp\" (UID: \"579e48ba-c588-4231-91ef-61c31ad241c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:37:52.261140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.260985    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5cb\" (UniqueName: \"kubernetes.io/projected/579e48ba-c588-4231-91ef-61c31ad241c4-kube-api-access-kc5cb\") pod \"kuadrant-operator-controller-manager-678d87766f-cvpvp\" (UID: \"579e48ba-c588-4231-91ef-61c31ad241c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:37:52.261355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.261335    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/579e48ba-c588-4231-91ef-61c31ad241c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cvpvp\" (UID: \"579e48ba-c588-4231-91ef-61c31ad241c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:37:52.326902 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.326828    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5cb\" (UniqueName: \"kubernetes.io/projected/579e48ba-c588-4231-91ef-61c31ad241c4-kube-api-access-kc5cb\") pod \"kuadrant-operator-controller-manager-678d87766f-cvpvp\" (UID: \"579e48ba-c588-4231-91ef-61c31ad241c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:37:52.332647 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.332625    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:37:52.349345 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.349322    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvcfb"
Nov 02 01:37:52.759739 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.759705    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"]
Nov 02 01:37:52.761558 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:37:52.761530    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579e48ba_c588_4231_91ef_61c31ad241c4.slice/crio-befde5e5a80e883226a570057ad787c197eea4ac27c6dd0f5ee59aeeb265bf65 WatchSource:0}: Error finding container befde5e5a80e883226a570057ad787c197eea4ac27c6dd0f5ee59aeeb265bf65: Status 404 returned error can't find the container with id befde5e5a80e883226a570057ad787c197eea4ac27c6dd0f5ee59aeeb265bf65
Nov 02 01:37:52.895360 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:52.895312    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:53.073438 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:53.073342    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:53.430651 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:53.430616    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp" event={"ID":"579e48ba-c588-4231-91ef-61c31ad241c4","Type":"ContainerStarted","Data":"af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df"}
Nov 02 01:37:53.430878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:53.430678    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp" event={"ID":"579e48ba-c588-4231-91ef-61c31ad241c4","Type":"ContainerStarted","Data":"befde5e5a80e883226a570057ad787c197eea4ac27c6dd0f5ee59aeeb265bf65"}
Nov 02 01:37:53.431472 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:53.431445    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:37:53.541695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:53.541663    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:53.558908 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:53.558857    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp" podStartSLOduration=2.558843667 podStartE2EDuration="2.558843667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:53.558484399 +0000 UTC m=+366.463949755" watchObservedRunningTime="2026-11-02 01:37:53.558843667 +0000 UTC m=+366.464309023"
Nov 02 01:37:53.723528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:53.723430    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:54.094401 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:54.094316    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:54.278631 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:54.278584    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:54.437462 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:54.437427    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq" event={"ID":"dd4641be-4555-471f-ba3a-ed2b4c7b9b8e","Type":"ContainerStarted","Data":"f61c1bff894c9e6d226eca786bcda77a5a14afc5fd0fd5270428304ba9ba0f60"}
Nov 02 01:37:54.655130 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:54.655076    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mfpzq" podStartSLOduration=1.812252506 podStartE2EDuration="4.655045844s" totalImagesPullingTime="2.842793338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:54.604691022 +0000 UTC m=+367.510156379" watchObservedRunningTime="2026-11-02 01:37:54.655045844 +0000 UTC m=+367.560511201"
Nov 02 01:37:56.480675 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:56.480641    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:56.719473 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:56.719442    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:58.307907 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:58.307873    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:58.391693 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:58.391657    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2hxrh"
Nov 02 01:37:59.159432 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:37:59.159382    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:00.409681 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:00.409654    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-jlqt6"
Nov 02 01:38:01.933818 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:01.933778    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:02.192115 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:02.192004    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:02.424988 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:02.424956    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j5pnp"
Nov 02 01:38:05.445502 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:05.445470    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 01:38:05.669363 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:05.669329    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"]
Nov 02 01:38:05.669580 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:05.669549    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42" podUID="70180f03-499e-439d-b3ba-ea7482b78cf2" containerName="manager" containerID="cri-o://e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642" gracePeriod=10
Nov 02 01:38:05.922828 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:05.922803    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:38:05.980033 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:05.980002    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70180f03-499e-439d-b3ba-ea7482b78cf2-kube-api-access-7kdhd\" (UniqueName: \"kubernetes.io/projected/70180f03-499e-439d-b3ba-ea7482b78cf2-kube-api-access-7kdhd\") pod \"70180f03-499e-439d-b3ba-ea7482b78cf2\" (UID: \"70180f03-499e-439d-b3ba-ea7482b78cf2\") "
Nov 02 01:38:05.980232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:05.980096    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70180f03-499e-439d-b3ba-ea7482b78cf2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70180f03-499e-439d-b3ba-ea7482b78cf2-extensions-socket-volume\") pod \"70180f03-499e-439d-b3ba-ea7482b78cf2\" (UID: \"70180f03-499e-439d-b3ba-ea7482b78cf2\") "
Nov 02 01:38:05.980456 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:05.980434    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70180f03-499e-439d-b3ba-ea7482b78cf2-extensions-socket-volume" pod "70180f03-499e-439d-b3ba-ea7482b78cf2" (UID: "70180f03-499e-439d-b3ba-ea7482b78cf2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:38:05.982475 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:05.982441    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70180f03-499e-439d-b3ba-ea7482b78cf2-kube-api-access-7kdhd" pod "70180f03-499e-439d-b3ba-ea7482b78cf2" (UID: "70180f03-499e-439d-b3ba-ea7482b78cf2"). InnerVolumeSpecName "kube-api-access-7kdhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:38:06.080930 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.080847    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kdhd\" (UniqueName: \"kubernetes.io/projected/70180f03-499e-439d-b3ba-ea7482b78cf2-kube-api-access-7kdhd\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:38:06.080930 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.080885    4912 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70180f03-499e-439d-b3ba-ea7482b78cf2-extensions-socket-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:38:06.363120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.363021    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:06.500609 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.500568    4912 generic.go:356] "Generic (PLEG): container finished" podID="70180f03-499e-439d-b3ba-ea7482b78cf2" containerID="e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642" exitCode=0
Nov 02 01:38:06.501005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.500620    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"
Nov 02 01:38:06.501005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.500654    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42" event={"ID":"70180f03-499e-439d-b3ba-ea7482b78cf2","Type":"ContainerDied","Data":"e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642"}
Nov 02 01:38:06.501005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.500723    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42" event={"ID":"70180f03-499e-439d-b3ba-ea7482b78cf2","Type":"ContainerDied","Data":"60bbc7d8879331cc387071b2aa238bc6c7d5c4f1aa677ba4935dbce2c00db845"}
Nov 02 01:38:06.501005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.500751    4912 scope.go:122] "RemoveContainer" containerID="e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642"
Nov 02 01:38:06.510668 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.510652    4912 scope.go:122] "RemoveContainer" containerID="e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642"
Nov 02 01:38:06.510939 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:38:06.510921    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642\": container with ID starting with e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642 not found: ID does not exist" containerID="e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642"
Nov 02 01:38:06.510979 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.510950    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642"} err="failed to get container status \"e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642\": rpc error: code = NotFound desc = could not find container \"e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642\": container with ID starting with e1b210d02b4502a6b924f26a91d9e076a91284629a19a8b8cce15c9ed6ce7642 not found: ID does not exist"
Nov 02 01:38:06.627847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.627808    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:06.667674 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.667641    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"]
Nov 02 01:38:06.674412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:06.674388    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cfd42"]
Nov 02 01:38:07.223149 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:07.223118    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70180f03-499e-439d-b3ba-ea7482b78cf2" path="/var/lib/kubelet/pods/70180f03-499e-439d-b3ba-ea7482b78cf2/volumes"
Nov 02 01:38:07.867245 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:07.867213    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:08.752398 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:08.752358    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:08.937389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:08.937351    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:09.894712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:09.894678    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:10.077859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:10.077822    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:11.978667 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:11.978630    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:12.140254 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:38:12.140212    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:38:12.209744 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:12.209709    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:12.476324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:12.476294    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:14.392856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:14.392825    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:14.568862 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:14.568821    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:14.943423 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:14.943385    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"]
Nov 02 01:38:14.943875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:14.943856    4912 cpu_manager.go:414] "containerMap: removing container" podUID="70180f03-499e-439d-b3ba-ea7482b78cf2" containerName="manager"
Nov 02 01:38:14.943978 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:14.943876    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70180f03-499e-439d-b3ba-ea7482b78cf2" containerName="manager"
Nov 02 01:38:14.943978 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:14.943960    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="70180f03-499e-439d-b3ba-ea7482b78cf2" containerName="manager"
Nov 02 01:38:14.947860 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:14.947832    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.000238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.000213    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jgdbk\""
Nov 02 01:38:15.051386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.051354    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"]
Nov 02 01:38:15.054186 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.054161    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj9pc\" (UniqueName: \"kubernetes.io/projected/4af3acb7-dc87-454d-b1de-70d65476d689-kube-api-access-wj9pc\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.054331 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.054202    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af3acb7-dc87-454d-b1de-70d65476d689-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.054331 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.054292    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.054331 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.054328    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.054515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.054348    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.054515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.054429    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.054515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.054457    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.054515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.054479    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af3acb7-dc87-454d-b1de-70d65476d689-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.054741 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.054565    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af3acb7-dc87-454d-b1de-70d65476d689-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.076219 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.076190    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:15.155239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155207    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af3acb7-dc87-454d-b1de-70d65476d689-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155408 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155257    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155475 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155449    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155513 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155499    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155568 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155553    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155608 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155598    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155644 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155622    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af3acb7-dc87-454d-b1de-70d65476d689-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155691 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155678    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155744 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155682    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af3acb7-dc87-454d-b1de-70d65476d689-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155805 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155744    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj9pc\" (UniqueName: \"kubernetes.io/projected/4af3acb7-dc87-454d-b1de-70d65476d689-kube-api-access-wj9pc\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155805 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155761    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.155913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.155845    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.156104 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.156052    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.156320 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.156301    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af3acb7-dc87-454d-b1de-70d65476d689-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.158099 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.158077    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af3acb7-dc87-454d-b1de-70d65476d689-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.158229 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.158211    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af3acb7-dc87-454d-b1de-70d65476d689-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.215990 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.215911    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af3acb7-dc87-454d-b1de-70d65476d689-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.216173 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.216131    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj9pc\" (UniqueName: \"kubernetes.io/projected/4af3acb7-dc87-454d-b1de-70d65476d689-kube-api-access-wj9pc\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485vxgl6\" (UID: \"4af3acb7-dc87-454d-b1de-70d65476d689\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.259067 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.259035    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:15.295528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.295489    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:15.682073 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.682023    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"]
Nov 02 01:38:15.683289 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:38:15.683262    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af3acb7_dc87_454d_b1de_70d65476d689.slice/crio-b8ddc77916ee0462cdaf89096799cd85fc868d9bb0b9a9b5976c646d3f6356ee WatchSource:0}: Error finding container b8ddc77916ee0462cdaf89096799cd85fc868d9bb0b9a9b5976c646d3f6356ee: Status 404 returned error can't find the container with id b8ddc77916ee0462cdaf89096799cd85fc868d9bb0b9a9b5976c646d3f6356ee
Nov 02 01:38:15.685443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.685414    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:38:15.685539 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.685478    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:38:15.685539 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:15.685512    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:38:16.562241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:16.562203    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6" event={"ID":"4af3acb7-dc87-454d-b1de-70d65476d689","Type":"ContainerStarted","Data":"2f426a96d7be72a03eba2910ea9a321fbd92064def8db8d4d6fc0ef352e31747"}
Nov 02 01:38:16.562445 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:16.562284    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6" event={"ID":"4af3acb7-dc87-454d-b1de-70d65476d689","Type":"ContainerStarted","Data":"b8ddc77916ee0462cdaf89096799cd85fc868d9bb0b9a9b5976c646d3f6356ee"}
Nov 02 01:38:16.730815 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:16.730765    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6" podStartSLOduration=2.730749094 podStartE2EDuration="2.730749094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:16.681171125 +0000 UTC m=+389.586636483" watchObservedRunningTime="2026-11-02 01:38:16.730749094 +0000 UTC m=+389.636214447"
Nov 02 01:38:17.255258 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.255225    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-bm5s5"]
Nov 02 01:38:17.257487 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.257469    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bm5s5"
Nov 02 01:38:17.259706 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.259682    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:17.264931 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.264784    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:17.309494 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.309471    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6m5m2\""
Nov 02 01:38:17.309701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.309683    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 01:38:17.309849 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.309835    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 01:38:17.315921 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.315892    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bm5s5"]
Nov 02 01:38:17.376601 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.376573    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83174cfa-b899-4192-9946-95a4a5447178-bound-sa-token\") pod \"cert-manager-597b95cb84-bm5s5\" (UID: \"83174cfa-b899-4192-9946-95a4a5447178\") " pod="cert-manager/cert-manager-597b95cb84-bm5s5"
Nov 02 01:38:17.376763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.376650    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9g4c\" (UniqueName: \"kubernetes.io/projected/83174cfa-b899-4192-9946-95a4a5447178-kube-api-access-w9g4c\") pod \"cert-manager-597b95cb84-bm5s5\" (UID: \"83174cfa-b899-4192-9946-95a4a5447178\") " pod="cert-manager/cert-manager-597b95cb84-bm5s5"
Nov 02 01:38:17.478037 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.478001    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9g4c\" (UniqueName: \"kubernetes.io/projected/83174cfa-b899-4192-9946-95a4a5447178-kube-api-access-w9g4c\") pod \"cert-manager-597b95cb84-bm5s5\" (UID: \"83174cfa-b899-4192-9946-95a4a5447178\") " pod="cert-manager/cert-manager-597b95cb84-bm5s5"
Nov 02 01:38:17.478249 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.478089    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83174cfa-b899-4192-9946-95a4a5447178-bound-sa-token\") pod \"cert-manager-597b95cb84-bm5s5\" (UID: \"83174cfa-b899-4192-9946-95a4a5447178\") " pod="cert-manager/cert-manager-597b95cb84-bm5s5"
Nov 02 01:38:17.538756 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.538676    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83174cfa-b899-4192-9946-95a4a5447178-bound-sa-token\") pod \"cert-manager-597b95cb84-bm5s5\" (UID: \"83174cfa-b899-4192-9946-95a4a5447178\") " pod="cert-manager/cert-manager-597b95cb84-bm5s5"
Nov 02 01:38:17.538918 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.538835    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9g4c\" (UniqueName: \"kubernetes.io/projected/83174cfa-b899-4192-9946-95a4a5447178-kube-api-access-w9g4c\") pod \"cert-manager-597b95cb84-bm5s5\" (UID: \"83174cfa-b899-4192-9946-95a4a5447178\") " pod="cert-manager/cert-manager-597b95cb84-bm5s5"
Nov 02 01:38:17.567204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.567174    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:17.568179 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.568159    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485vxgl6"
Nov 02 01:38:17.579550 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.579528    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bm5s5"
Nov 02 01:38:17.950358 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:17.950338    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bm5s5"]
Nov 02 01:38:17.952157 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:38:17.952128    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83174cfa_b899_4192_9946_95a4a5447178.slice/crio-bd5ce4503bb439db1374b9babc1b5bc8db18fd4b58da58ce63eccf2d3884a36b WatchSource:0}: Error finding container bd5ce4503bb439db1374b9babc1b5bc8db18fd4b58da58ce63eccf2d3884a36b: Status 404 returned error can't find the container with id bd5ce4503bb439db1374b9babc1b5bc8db18fd4b58da58ce63eccf2d3884a36b
Nov 02 01:38:18.573701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:18.573664    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bm5s5" event={"ID":"83174cfa-b899-4192-9946-95a4a5447178","Type":"ContainerStarted","Data":"bd5ce4503bb439db1374b9babc1b5bc8db18fd4b58da58ce63eccf2d3884a36b"}
Nov 02 01:38:22.076810 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:22.076780    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:23.601901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:23.601802    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bm5s5" event={"ID":"83174cfa-b899-4192-9946-95a4a5447178","Type":"ContainerStarted","Data":"2b9ef1fe39eda460d7d7678babf490e4026141307587b0f8e1ee8934cf8ba40a"}
Nov 02 01:38:23.718043 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:23.717988    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-bm5s5" podStartSLOduration=1.54702396 podStartE2EDuration="6.717973065s" totalImagesPullingTime="5.170949105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:23.716343944 +0000 UTC m=+396.621809309" watchObservedRunningTime="2026-11-02 01:38:23.717973065 +0000 UTC m=+396.623438422"
Nov 02 01:38:25.977263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:25.977211    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:26.154285 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.154250    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:26.327232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.327139    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:26.854801 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.854765    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"]
Nov 02 01:38:26.859559 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.859536    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:26.911798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.911742    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 01:38:26.911798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.911743    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 01:38:26.911798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.911803    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-5vctx\""
Nov 02 01:38:26.963218 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.963187    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34c90cfc-6bba-478f-b339-afff5290ffa0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:26.963410 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.963226    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34c90cfc-6bba-478f-b339-afff5290ffa0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:26.963410 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.963355    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34c90cfc-6bba-478f-b339-afff5290ffa0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:26.963533 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:26.963476    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5x45\" (UniqueName: \"kubernetes.io/projected/34c90cfc-6bba-478f-b339-afff5290ffa0-kube-api-access-p5x45\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.061140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.061105    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 01:38:27.063741 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.063711    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"]
Nov 02 01:38:27.063942 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.063916    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5x45\" (UniqueName: \"kubernetes.io/projected/34c90cfc-6bba-478f-b339-afff5290ffa0-kube-api-access-p5x45\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.064000 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.063964    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34c90cfc-6bba-478f-b339-afff5290ffa0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.064038 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.063996    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34c90cfc-6bba-478f-b339-afff5290ffa0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.064106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.064049    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34c90cfc-6bba-478f-b339-afff5290ffa0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.064475 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.064446    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34c90cfc-6bba-478f-b339-afff5290ffa0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.064535 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.064456    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34c90cfc-6bba-478f-b339-afff5290ffa0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.064913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.064893    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34c90cfc-6bba-478f-b339-afff5290ffa0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.125022 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.124994    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5x45\" (UniqueName: \"kubernetes.io/projected/34c90cfc-6bba-478f-b339-afff5290ffa0-kube-api-access-p5x45\") pod \"kserve-module-controller-manager-7566cf5fc8-xgnbh\" (UID: \"34c90cfc-6bba-478f-b339-afff5290ffa0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.171594 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.171558    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:27.192106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.192046    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:27.451568 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.451486    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:27.605088 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.605043    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"]
Nov 02 01:38:27.606708 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:38:27.606681    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c90cfc_6bba_478f_b339_afff5290ffa0.slice/crio-5708223f145e81d66e47838fe087113f3ec9ed45aa9efa596cc441e44ae5f85f WatchSource:0}: Error finding container 5708223f145e81d66e47838fe087113f3ec9ed45aa9efa596cc441e44ae5f85f: Status 404 returned error can't find the container with id 5708223f145e81d66e47838fe087113f3ec9ed45aa9efa596cc441e44ae5f85f
Nov 02 01:38:27.624339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.624232    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh" event={"ID":"34c90cfc-6bba-478f-b339-afff5290ffa0","Type":"ContainerStarted","Data":"5708223f145e81d66e47838fe087113f3ec9ed45aa9efa596cc441e44ae5f85f"}
Nov 02 01:38:27.630381 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:27.630353    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:28.621289 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:28.621249    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lddw4"]
Nov 02 01:38:28.624998 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:28.624972    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lddw4"
Nov 02 01:38:28.676820 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:28.676763    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np7hn\" (UniqueName: \"kubernetes.io/projected/990f28bb-2de3-4eab-80f0-db32ad6d966a-kube-api-access-np7hn\") pod \"authorino-57f7f8659f-lddw4\" (UID: \"990f28bb-2de3-4eab-80f0-db32ad6d966a\") " pod="kuadrant-system/authorino-57f7f8659f-lddw4"
Nov 02 01:38:28.677572 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:28.677547    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-799rl\""
Nov 02 01:38:28.678368 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:28.678346    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lddw4"]
Nov 02 01:38:28.777854 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:28.777819    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np7hn\" (UniqueName: \"kubernetes.io/projected/990f28bb-2de3-4eab-80f0-db32ad6d966a-kube-api-access-np7hn\") pod \"authorino-57f7f8659f-lddw4\" (UID: \"990f28bb-2de3-4eab-80f0-db32ad6d966a\") " pod="kuadrant-system/authorino-57f7f8659f-lddw4"
Nov 02 01:38:28.816576 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:28.816533    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:28.839032 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:28.838994    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np7hn\" (UniqueName: \"kubernetes.io/projected/990f28bb-2de3-4eab-80f0-db32ad6d966a-kube-api-access-np7hn\") pod \"authorino-57f7f8659f-lddw4\" (UID: \"990f28bb-2de3-4eab-80f0-db32ad6d966a\") " pod="kuadrant-system/authorino-57f7f8659f-lddw4"
Nov 02 01:38:28.940780 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:28.940708    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lddw4"
Nov 02 01:38:29.043245 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:29.043147    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:29.324072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:29.324029    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lddw4"]
Nov 02 01:38:29.326602 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:38:29.326567    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod990f28bb_2de3_4eab_80f0_db32ad6d966a.slice/crio-1c1da0c093ea77d60a6707d93208d0d50ba6f4a578bc491d31b79d1408842c1c WatchSource:0}: Error finding container 1c1da0c093ea77d60a6707d93208d0d50ba6f4a578bc491d31b79d1408842c1c: Status 404 returned error can't find the container with id 1c1da0c093ea77d60a6707d93208d0d50ba6f4a578bc491d31b79d1408842c1c
Nov 02 01:38:29.636677 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:29.636637    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lddw4" event={"ID":"990f28bb-2de3-4eab-80f0-db32ad6d966a","Type":"ContainerStarted","Data":"1c1da0c093ea77d60a6707d93208d0d50ba6f4a578bc491d31b79d1408842c1c"}
Nov 02 01:38:31.651111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:31.651077    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh" event={"ID":"34c90cfc-6bba-478f-b339-afff5290ffa0","Type":"ContainerStarted","Data":"24e9cc247b7cb99d0553ef395cee4d1ae307738715da07c04b91b724fc284db2"}
Nov 02 01:38:31.651629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:31.651154    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:31.818824 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:31.818764    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh" podStartSLOduration=2.369206591 podStartE2EDuration="5.818745884s" totalImagesPullingTime="3.449539293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:31.818159644 +0000 UTC m=+404.723625001" watchObservedRunningTime="2026-11-02 01:38:31.818745884 +0000 UTC m=+404.724211243"
Nov 02 01:38:33.669780 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:33.669735    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lddw4" event={"ID":"990f28bb-2de3-4eab-80f0-db32ad6d966a","Type":"ContainerStarted","Data":"7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf"}
Nov 02 01:38:33.783680 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:33.783623    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-lddw4" podStartSLOduration=1.914058957 podStartE2EDuration="5.783609222s" totalImagesPullingTime="3.869550265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:33.781846963 +0000 UTC m=+406.687312339" watchObservedRunningTime="2026-11-02 01:38:33.783609222 +0000 UTC m=+406.689074579"
Nov 02 01:38:35.401487 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:35.401449    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lddw4"]
Nov 02 01:38:35.680626 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:35.680534    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-lddw4" podUID="990f28bb-2de3-4eab-80f0-db32ad6d966a" containerName="authorino" containerID="cri-o://7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf" gracePeriod=30
Nov 02 01:38:35.928954 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:35.928924    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lddw4"
Nov 02 01:38:36.046383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.046306    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990f28bb-2de3-4eab-80f0-db32ad6d966a-kube-api-access-np7hn\" (UniqueName: \"kubernetes.io/projected/990f28bb-2de3-4eab-80f0-db32ad6d966a-kube-api-access-np7hn\") pod \"990f28bb-2de3-4eab-80f0-db32ad6d966a\" (UID: \"990f28bb-2de3-4eab-80f0-db32ad6d966a\") "
Nov 02 01:38:36.048957 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.048917    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990f28bb-2de3-4eab-80f0-db32ad6d966a-kube-api-access-np7hn" pod "990f28bb-2de3-4eab-80f0-db32ad6d966a" (UID: "990f28bb-2de3-4eab-80f0-db32ad6d966a"). InnerVolumeSpecName "kube-api-access-np7hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:38:36.147772 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.147714    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np7hn\" (UniqueName: \"kubernetes.io/projected/990f28bb-2de3-4eab-80f0-db32ad6d966a-kube-api-access-np7hn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:38:36.687157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.687122    4912 generic.go:356] "Generic (PLEG): container finished" podID="990f28bb-2de3-4eab-80f0-db32ad6d966a" containerID="7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf" exitCode=0
Nov 02 01:38:36.687157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.687158    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lddw4"
Nov 02 01:38:36.687720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.687206    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lddw4" event={"ID":"990f28bb-2de3-4eab-80f0-db32ad6d966a","Type":"ContainerDied","Data":"7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf"}
Nov 02 01:38:36.687720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.687252    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lddw4" event={"ID":"990f28bb-2de3-4eab-80f0-db32ad6d966a","Type":"ContainerDied","Data":"1c1da0c093ea77d60a6707d93208d0d50ba6f4a578bc491d31b79d1408842c1c"}
Nov 02 01:38:36.687720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.687276    4912 scope.go:122] "RemoveContainer" containerID="7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf"
Nov 02 01:38:36.697200 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.697177    4912 scope.go:122] "RemoveContainer" containerID="7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf"
Nov 02 01:38:36.697510 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:38:36.697490    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf\": container with ID starting with 7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf not found: ID does not exist" containerID="7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf"
Nov 02 01:38:36.697587 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.697524    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf"} err="failed to get container status \"7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf\": rpc error: code = NotFound desc = could not find container \"7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf\": container with ID starting with 7bddb53c56237556365beac65fd45d4f2c0acfda9ce37331d49a213d0ab51ebf not found: ID does not exist"
Nov 02 01:38:36.850466 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.850432    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lddw4"]
Nov 02 01:38:36.900017 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:36.899984    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lddw4"]
Nov 02 01:38:37.224118 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:37.224076    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990f28bb-2de3-4eab-80f0-db32ad6d966a" path="/var/lib/kubelet/pods/990f28bb-2de3-4eab-80f0-db32ad6d966a/volumes"
Nov 02 01:38:39.798453 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:39.798418    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:41.260076 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:41.260030    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:42.137306 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:38:42.137270    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:38:49.857619 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:49.857586    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:50.042430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.042388    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:50.491637 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.491604    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"]
Nov 02 01:38:50.492047 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.492033    4912 cpu_manager.go:414] "containerMap: removing container" podUID="990f28bb-2de3-4eab-80f0-db32ad6d966a" containerName="authorino"
Nov 02 01:38:50.492121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.492048    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990f28bb-2de3-4eab-80f0-db32ad6d966a" containerName="authorino"
Nov 02 01:38:50.492157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.492133    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="990f28bb-2de3-4eab-80f0-db32ad6d966a" containerName="authorino"
Nov 02 01:38:50.496801 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.496779    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:38:50.549829 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.549799    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:38:50.550131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.549896    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:38:50.599398 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.599365    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 02 01:38:50.599602 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.599491    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-m2j64\""
Nov 02 01:38:50.601782 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.601760    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"]
Nov 02 01:38:50.605132 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.605111    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:50.687006 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.686968    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4e728077-b906-4311-a8d3-15ebb7d56238-data\") pod \"postgres-7cdbf6dc44-6vlcv\" (UID: \"4e728077-b906-4311-a8d3-15ebb7d56238\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:38:50.687259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.687157    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hrwg\" (UniqueName: \"kubernetes.io/projected/4e728077-b906-4311-a8d3-15ebb7d56238-kube-api-access-7hrwg\") pod \"postgres-7cdbf6dc44-6vlcv\" (UID: \"4e728077-b906-4311-a8d3-15ebb7d56238\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:38:50.787659 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.787563    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hrwg\" (UniqueName: \"kubernetes.io/projected/4e728077-b906-4311-a8d3-15ebb7d56238-kube-api-access-7hrwg\") pod \"postgres-7cdbf6dc44-6vlcv\" (UID: \"4e728077-b906-4311-a8d3-15ebb7d56238\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:38:50.787659 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.787630    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4e728077-b906-4311-a8d3-15ebb7d56238-data\") pod \"postgres-7cdbf6dc44-6vlcv\" (UID: \"4e728077-b906-4311-a8d3-15ebb7d56238\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:38:50.788137 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.788109    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4e728077-b906-4311-a8d3-15ebb7d56238-data\") pod \"postgres-7cdbf6dc44-6vlcv\" (UID: \"4e728077-b906-4311-a8d3-15ebb7d56238\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:38:50.849166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:50.849125    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hrwg\" (UniqueName: \"kubernetes.io/projected/4e728077-b906-4311-a8d3-15ebb7d56238-kube-api-access-7hrwg\") pod \"postgres-7cdbf6dc44-6vlcv\" (UID: \"4e728077-b906-4311-a8d3-15ebb7d56238\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:38:51.110519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:51.110413    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:38:51.483628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:51.483597    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"]
Nov 02 01:38:51.485776 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:38:51.485749    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e728077_b906_4311_a8d3_15ebb7d56238.slice/crio-fef05a56f2029701ea46dd85a3f04d9ab04fdb694113e1ae267d39c3f2d22ddf WatchSource:0}: Error finding container fef05a56f2029701ea46dd85a3f04d9ab04fdb694113e1ae267d39c3f2d22ddf: Status 404 returned error can't find the container with id fef05a56f2029701ea46dd85a3f04d9ab04fdb694113e1ae267d39c3f2d22ddf
Nov 02 01:38:51.778167 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:51.778085    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv" event={"ID":"4e728077-b906-4311-a8d3-15ebb7d56238","Type":"ContainerStarted","Data":"fef05a56f2029701ea46dd85a3f04d9ab04fdb694113e1ae267d39c3f2d22ddf"}
Nov 02 01:38:52.657679 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:52.657655    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xgnbh"
Nov 02 01:38:58.820829 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:58.820795    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv" event={"ID":"4e728077-b906-4311-a8d3-15ebb7d56238","Type":"ContainerStarted","Data":"0f736652694c583b4348527c33818e44724822067bee01757bb4644ead60e247"}
Nov 02 01:38:58.821263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:58.820961    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:38:58.936900 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:38:58.936850    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv" podStartSLOduration=2.243366364 podStartE2EDuration="8.936835994s" totalImagesPullingTime="6.69346963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:58.934930307 +0000 UTC m=+431.840395665" watchObservedRunningTime="2026-11-02 01:38:58.936835994 +0000 UTC m=+431.842301351"
Nov 02 01:39:04.856709 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:04.856677    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6vlcv"
Nov 02 01:39:11.950434 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:11.950393    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:12.138096 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:39:12.138034    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:39:12.178687 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:12.178638    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:12.354019 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:12.353938    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:12.532608 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:12.532574    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:16.621497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:16.621462    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:16.865696 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:16.865664    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:19.651761 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.651726    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"]
Nov 02 01:39:19.660039 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.660019    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.726003 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.725975    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 02 01:39:19.726003 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.725975    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 02 01:39:19.741874 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.741845    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/781242d4-1321-4bb2-8c4d-e27af6cc95c6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.742033 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.741992    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwlq\" (UniqueName: \"kubernetes.io/projected/781242d4-1321-4bb2-8c4d-e27af6cc95c6-kube-api-access-lmwlq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.742033 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.742027    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/781242d4-1321-4bb2-8c4d-e27af6cc95c6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.742162 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.742143    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/781242d4-1321-4bb2-8c4d-e27af6cc95c6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.775749 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.775725    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 02 01:39:19.775876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.775825    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-24cgz\""
Nov 02 01:39:19.778106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.778087    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"]
Nov 02 01:39:19.842852 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.842820    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/781242d4-1321-4bb2-8c4d-e27af6cc95c6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.843034 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.842883    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwlq\" (UniqueName: \"kubernetes.io/projected/781242d4-1321-4bb2-8c4d-e27af6cc95c6-kube-api-access-lmwlq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.843034 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.842901    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/781242d4-1321-4bb2-8c4d-e27af6cc95c6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.843034 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.842933    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/781242d4-1321-4bb2-8c4d-e27af6cc95c6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.843034 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:39:19.842953    4912 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 02 01:39:19.843034 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:39:19.843035    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/781242d4-1321-4bb2-8c4d-e27af6cc95c6-proxy-source podName:781242d4-1321-4bb2-8c4d-e27af6cc95c6 nodeName:}" failed. No retries permitted until 2026-11-02 01:39:20.343011163 +0000 UTC m=+453.248476500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/781242d4-1321-4bb2-8c4d-e27af6cc95c6-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-7bpsx" (UID: "781242d4-1321-4bb2-8c4d-e27af6cc95c6") : configmap "usage-tenancy-proxy-source" not found
Nov 02 01:39:19.843284 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:39:19.843084    4912 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 02 01:39:19.843284 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:39:19.843147    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/781242d4-1321-4bb2-8c4d-e27af6cc95c6-tls-cert podName:781242d4-1321-4bb2-8c4d-e27af6cc95c6 nodeName:}" failed. No retries permitted until 2026-11-02 01:39:20.343130387 +0000 UTC m=+453.248595723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/781242d4-1321-4bb2-8c4d-e27af6cc95c6-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-7bpsx" (UID: "781242d4-1321-4bb2-8c4d-e27af6cc95c6") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 02 01:39:19.843620 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.843593    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/781242d4-1321-4bb2-8c4d-e27af6cc95c6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:19.880958 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.880918    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:19.905959 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:19.905883    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwlq\" (UniqueName: \"kubernetes.io/projected/781242d4-1321-4bb2-8c4d-e27af6cc95c6-kube-api-access-lmwlq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:20.168275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:20.168175    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:20.348021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:20.347985    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/781242d4-1321-4bb2-8c4d-e27af6cc95c6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:20.348217 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:20.348093    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/781242d4-1321-4bb2-8c4d-e27af6cc95c6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:20.348974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:20.348932    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/781242d4-1321-4bb2-8c4d-e27af6cc95c6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:20.350749 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:20.350723    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/781242d4-1321-4bb2-8c4d-e27af6cc95c6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7bpsx\" (UID: \"781242d4-1321-4bb2-8c4d-e27af6cc95c6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:20.573993 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:20.573909    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:20.955007 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:20.954979    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"]
Nov 02 01:39:20.957267 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:39:20.957241    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781242d4_1321_4bb2_8c4d_e27af6cc95c6.slice/crio-f71fe942a7be6a6a1917782b16057bd715eb68978106d3e417e48665d94b666e WatchSource:0}: Error finding container f71fe942a7be6a6a1917782b16057bd715eb68978106d3e417e48665d94b666e: Status 404 returned error can't find the container with id f71fe942a7be6a6a1917782b16057bd715eb68978106d3e417e48665d94b666e
Nov 02 01:39:21.951580 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:21.951539    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx" event={"ID":"781242d4-1321-4bb2-8c4d-e27af6cc95c6","Type":"ContainerStarted","Data":"f71fe942a7be6a6a1917782b16057bd715eb68978106d3e417e48665d94b666e"}
Nov 02 01:39:33.778962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:33.778925    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:33.959429 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:33.959396    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:34.145797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.145757    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:34.325418 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.325379    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:34.506988 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.506897    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:34.656860 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.656820    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dpzkp"]
Nov 02 01:39:34.676812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.676779    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:39:34.684985 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.684954    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:34.707019 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.706989    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dpzkp"]
Nov 02 01:39:34.784742 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.784662    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f61dd1-d6b3-4ac6-8f79-2a241265c803-cert\") pod \"kserve-controller-manager-776cc66d47-dpzkp\" (UID: \"c0f61dd1-d6b3-4ac6-8f79-2a241265c803\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:39:34.784742 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.784730    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btwwg\" (UniqueName: \"kubernetes.io/projected/c0f61dd1-d6b3-4ac6-8f79-2a241265c803-kube-api-access-btwwg\") pod \"kserve-controller-manager-776cc66d47-dpzkp\" (UID: \"c0f61dd1-d6b3-4ac6-8f79-2a241265c803\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:39:34.798782 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.798733    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-h4wnk\""
Nov 02 01:39:34.798782 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.798765    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 01:39:34.805733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.805710    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"]
Nov 02 01:39:34.809520 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.809492    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:39:34.860110 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.860052    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:34.886133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.886095    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ac4161e-a1fc-40e4-8e87-46b77007d077-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5vwsx\" (UID: \"1ac4161e-a1fc-40e4-8e87-46b77007d077\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:39:34.886446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.886417    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f61dd1-d6b3-4ac6-8f79-2a241265c803-cert\") pod \"kserve-controller-manager-776cc66d47-dpzkp\" (UID: \"c0f61dd1-d6b3-4ac6-8f79-2a241265c803\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:39:34.886580 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.886487    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vtvb\" (UniqueName: \"kubernetes.io/projected/1ac4161e-a1fc-40e4-8e87-46b77007d077-kube-api-access-9vtvb\") pod \"llmisvc-controller-manager-869d5dcc5d-5vwsx\" (UID: \"1ac4161e-a1fc-40e4-8e87-46b77007d077\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:39:34.886580 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.886530    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btwwg\" (UniqueName: \"kubernetes.io/projected/c0f61dd1-d6b3-4ac6-8f79-2a241265c803-kube-api-access-btwwg\") pod \"kserve-controller-manager-776cc66d47-dpzkp\" (UID: \"c0f61dd1-d6b3-4ac6-8f79-2a241265c803\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:39:34.889152 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.889131    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f61dd1-d6b3-4ac6-8f79-2a241265c803-cert\") pod \"kserve-controller-manager-776cc66d47-dpzkp\" (UID: \"c0f61dd1-d6b3-4ac6-8f79-2a241265c803\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:39:34.900654 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.900633    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"]
Nov 02 01:39:34.904783 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.904767    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 02 01:39:34.927131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.927099    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jxj4b\""
Nov 02 01:39:34.950627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.950599    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-q49dp"]
Nov 02 01:39:34.954764 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.954742    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:34.962118 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.962093    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btwwg\" (UniqueName: \"kubernetes.io/projected/c0f61dd1-d6b3-4ac6-8f79-2a241265c803-kube-api-access-btwwg\") pod \"kserve-controller-manager-776cc66d47-dpzkp\" (UID: \"c0f61dd1-d6b3-4ac6-8f79-2a241265c803\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:39:34.987518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.987488    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ac4161e-a1fc-40e4-8e87-46b77007d077-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5vwsx\" (UID: \"1ac4161e-a1fc-40e4-8e87-46b77007d077\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:39:34.987674 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.987561    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vtvb\" (UniqueName: \"kubernetes.io/projected/1ac4161e-a1fc-40e4-8e87-46b77007d077-kube-api-access-9vtvb\") pod \"llmisvc-controller-manager-869d5dcc5d-5vwsx\" (UID: \"1ac4161e-a1fc-40e4-8e87-46b77007d077\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:39:34.991416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.991392    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ac4161e-a1fc-40e4-8e87-46b77007d077-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5vwsx\" (UID: \"1ac4161e-a1fc-40e4-8e87-46b77007d077\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:39:34.992206 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:34.992188    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:39:35.000839 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.000803    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-q49dp"]
Nov 02 01:39:35.026353 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.026326    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 02 01:39:35.058482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.058429    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hqsp5\""
Nov 02 01:39:35.089082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.088443    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb49p\" (UniqueName: \"kubernetes.io/projected/dc1dccb6-466f-4fba-85fb-528e41a92d87-kube-api-access-bb49p\") pod \"model-serving-api-75bd89fdfc-q49dp\" (UID: \"dc1dccb6-466f-4fba-85fb-528e41a92d87\") " pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:35.089082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.088478    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1dccb6-466f-4fba-85fb-528e41a92d87-tls-certs\") pod \"model-serving-api-75bd89fdfc-q49dp\" (UID: \"dc1dccb6-466f-4fba-85fb-528e41a92d87\") " pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:35.099692 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.099660    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vtvb\" (UniqueName: \"kubernetes.io/projected/1ac4161e-a1fc-40e4-8e87-46b77007d077-kube-api-access-9vtvb\") pod \"llmisvc-controller-manager-869d5dcc5d-5vwsx\" (UID: \"1ac4161e-a1fc-40e4-8e87-46b77007d077\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:39:35.125521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.125477    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:39:35.190901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.189699    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb49p\" (UniqueName: \"kubernetes.io/projected/dc1dccb6-466f-4fba-85fb-528e41a92d87-kube-api-access-bb49p\") pod \"model-serving-api-75bd89fdfc-q49dp\" (UID: \"dc1dccb6-466f-4fba-85fb-528e41a92d87\") " pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:35.190901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.189752    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1dccb6-466f-4fba-85fb-528e41a92d87-tls-certs\") pod \"model-serving-api-75bd89fdfc-q49dp\" (UID: \"dc1dccb6-466f-4fba-85fb-528e41a92d87\") " pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:35.193579 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.193527    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1dccb6-466f-4fba-85fb-528e41a92d87-tls-certs\") pod \"model-serving-api-75bd89fdfc-q49dp\" (UID: \"dc1dccb6-466f-4fba-85fb-528e41a92d87\") " pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:35.262863 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.262837    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb49p\" (UniqueName: \"kubernetes.io/projected/dc1dccb6-466f-4fba-85fb-528e41a92d87-kube-api-access-bb49p\") pod \"model-serving-api-75bd89fdfc-q49dp\" (UID: \"dc1dccb6-466f-4fba-85fb-528e41a92d87\") " pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:35.280663 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.280630    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:35.377103 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.377076    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dpzkp"]
Nov 02 01:39:35.379041 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:39:35.379008    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f61dd1_d6b3_4ac6_8f79_2a241265c803.slice/crio-b9a7237c8d029a4fd337db378ec7926c81fe42d645c231ebe868f0c83754666d WatchSource:0}: Error finding container b9a7237c8d029a4fd337db378ec7926c81fe42d645c231ebe868f0c83754666d: Status 404 returned error can't find the container with id b9a7237c8d029a4fd337db378ec7926c81fe42d645c231ebe868f0c83754666d
Nov 02 01:39:35.501112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.501086    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"]
Nov 02 01:39:35.503679 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:39:35.503649    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1ac4161e_a1fc_40e4_8e87_46b77007d077.slice/crio-6371be06397a0b2cd3b0f3aac56b3c0511ef088a4669be9455e49fd36056982c WatchSource:0}: Error finding container 6371be06397a0b2cd3b0f3aac56b3c0511ef088a4669be9455e49fd36056982c: Status 404 returned error can't find the container with id 6371be06397a0b2cd3b0f3aac56b3c0511ef088a4669be9455e49fd36056982c
Nov 02 01:39:35.660626 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:35.660603    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-q49dp"]
Nov 02 01:39:35.662628 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:39:35.662600    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1dccb6_466f_4fba_85fb_528e41a92d87.slice/crio-696ea2d9ee02f93917b63fc7e8e54420185899f34b4b622cccbdab3b75dde42d WatchSource:0}: Error finding container 696ea2d9ee02f93917b63fc7e8e54420185899f34b4b622cccbdab3b75dde42d: Status 404 returned error can't find the container with id 696ea2d9ee02f93917b63fc7e8e54420185899f34b4b622cccbdab3b75dde42d
Nov 02 01:39:36.045871 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:36.045779    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp" event={"ID":"c0f61dd1-d6b3-4ac6-8f79-2a241265c803","Type":"ContainerStarted","Data":"b9a7237c8d029a4fd337db378ec7926c81fe42d645c231ebe868f0c83754666d"}
Nov 02 01:39:36.047371 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:36.047343    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx" event={"ID":"1ac4161e-a1fc-40e4-8e87-46b77007d077","Type":"ContainerStarted","Data":"6371be06397a0b2cd3b0f3aac56b3c0511ef088a4669be9455e49fd36056982c"}
Nov 02 01:39:36.048692 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:36.048663    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-q49dp" event={"ID":"dc1dccb6-466f-4fba-85fb-528e41a92d87","Type":"ContainerStarted","Data":"696ea2d9ee02f93917b63fc7e8e54420185899f34b4b622cccbdab3b75dde42d"}
Nov 02 01:39:36.072282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:36.072239    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:39.080429 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:39.080339    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx" event={"ID":"1ac4161e-a1fc-40e4-8e87-46b77007d077","Type":"ContainerStarted","Data":"e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003"}
Nov 02 01:39:39.080799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:39.080442    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:39:39.082514 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:39.082412    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-q49dp" event={"ID":"dc1dccb6-466f-4fba-85fb-528e41a92d87","Type":"ContainerStarted","Data":"07b5a08ef305d1ec5c1d1239258a3f36365a3689a3c5567564733fe37e51c082"}
Nov 02 01:39:39.082514 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:39.082473    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:39.084219 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:39.084190    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp" event={"ID":"c0f61dd1-d6b3-4ac6-8f79-2a241265c803","Type":"ContainerStarted","Data":"6ff0f6e5b3b43e577759dc5aea83a660245252cfd28ade1c45e18cac863c4923"}
Nov 02 01:39:39.084350 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:39.084255    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:39:39.245808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:39.245752    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx" podStartSLOduration=2.009845095 podStartE2EDuration="5.245738944s" totalImagesPullingTime="3.235893849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:39.195741839 +0000 UTC m=+472.101207195" watchObservedRunningTime="2026-11-02 01:39:39.245738944 +0000 UTC m=+472.151204301"
Nov 02 01:39:39.318281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:39.318226    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-q49dp" podStartSLOduration=2.462598839 podStartE2EDuration="5.318211335s" totalImagesPullingTime="2.855612496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:39.316311747 +0000 UTC m=+472.221777116" watchObservedRunningTime="2026-11-02 01:39:39.318211335 +0000 UTC m=+472.223676728"
Nov 02 01:39:39.435224 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:39.435178    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp" podStartSLOduration=2.381973289 podStartE2EDuration="5.435164024s" totalImagesPullingTime="3.053190735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:39.431873123 +0000 UTC m=+472.337338480" watchObservedRunningTime="2026-11-02 01:39:39.435164024 +0000 UTC m=+472.340629381"
Nov 02 01:39:40.089655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:40.089573    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:40.091573 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:40.091548    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx" event={"ID":"781242d4-1321-4bb2-8c4d-e27af6cc95c6","Type":"ContainerStarted","Data":"dd2ca93e260d06487c53aa6c8551b004e21581b88e4e11ccf5dca900f6dfb048"}
Nov 02 01:39:40.091726 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:40.091624    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:40.256720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:40.256669    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx" podStartSLOduration=2.5646868449999998 podStartE2EDuration="21.256656355s" totalImagesPullingTime="18.69196951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:40.206978443 +0000 UTC m=+473.112443797" watchObservedRunningTime="2026-11-02 01:39:40.256656355 +0000 UTC m=+473.162121712"
Nov 02 01:39:41.287771 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:41.287736    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:41.466676 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:41.466649    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:41.640543 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:41.640504    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:41.819489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:41.819445    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:41.997252 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:41.997170    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:42.136839 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:39:42.136795    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:39:42.179504 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:42.179477    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:42.358889 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:42.358813    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:42.542877 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:42.542835    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:47.097677 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:47.097637    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:49.887568 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:49.887532    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:50.064886 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:50.064846    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:50.096176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:50.096145    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-q49dp"
Nov 02 01:39:51.100969 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:51.100932    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7bpsx"
Nov 02 01:39:51.984356 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:51.984314    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:52.211368 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:52.211341    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:39:52.396299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:39:52.396268    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:40:04.188209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.188127    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:40:04.440003 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.439931    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrbrm"]
Nov 02 01:40:04.449472 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.449436    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:40:04.453497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.453476    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm"
Nov 02 01:40:04.499047 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.499024    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrbrm"]
Nov 02 01:40:04.540553 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.540525    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wkb\" (UniqueName: \"kubernetes.io/projected/efebec03-c7c7-4607-895b-fd7bb1b6a925-kube-api-access-g2wkb\") pod \"authorino-6f6944bcdd-lrbrm\" (UID: \"efebec03-c7c7-4607-895b-fd7bb1b6a925\") " pod="kuadrant-system/authorino-6f6944bcdd-lrbrm"
Nov 02 01:40:04.545851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.545830    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-799rl\""
Nov 02 01:40:04.641715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.641681    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wkb\" (UniqueName: \"kubernetes.io/projected/efebec03-c7c7-4607-895b-fd7bb1b6a925-kube-api-access-g2wkb\") pod \"authorino-6f6944bcdd-lrbrm\" (UID: \"efebec03-c7c7-4607-895b-fd7bb1b6a925\") " pod="kuadrant-system/authorino-6f6944bcdd-lrbrm"
Nov 02 01:40:04.702577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.702503    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wkb\" (UniqueName: \"kubernetes.io/projected/efebec03-c7c7-4607-895b-fd7bb1b6a925-kube-api-access-g2wkb\") pod \"authorino-6f6944bcdd-lrbrm\" (UID: \"efebec03-c7c7-4607-895b-fd7bb1b6a925\") " pod="kuadrant-system/authorino-6f6944bcdd-lrbrm"
Nov 02 01:40:04.764831 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.764790    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm"
Nov 02 01:40:04.785397 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.785369    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrbrm"]
Nov 02 01:40:04.814788 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.814739    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-757c8fdd8-w4knj"]
Nov 02 01:40:04.817820 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.817797    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757c8fdd8-w4knj"
Nov 02 01:40:04.844810 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.844385    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6p7w\" (UniqueName: \"kubernetes.io/projected/52e40a04-8a80-476b-b34c-b4d18db25eb9-kube-api-access-w6p7w\") pod \"authorino-757c8fdd8-w4knj\" (UID: \"52e40a04-8a80-476b-b34c-b4d18db25eb9\") " pod="kuadrant-system/authorino-757c8fdd8-w4knj"
Nov 02 01:40:04.874490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.874456    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-757c8fdd8-w4knj"]
Nov 02 01:40:04.945903 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:04.945858    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6p7w\" (UniqueName: \"kubernetes.io/projected/52e40a04-8a80-476b-b34c-b4d18db25eb9-kube-api-access-w6p7w\") pod \"authorino-757c8fdd8-w4knj\" (UID: \"52e40a04-8a80-476b-b34c-b4d18db25eb9\") " pod="kuadrant-system/authorino-757c8fdd8-w4knj"
Nov 02 01:40:05.007095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.006982    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6p7w\" (UniqueName: \"kubernetes.io/projected/52e40a04-8a80-476b-b34c-b4d18db25eb9-kube-api-access-w6p7w\") pod \"authorino-757c8fdd8-w4knj\" (UID: \"52e40a04-8a80-476b-b34c-b4d18db25eb9\") " pod="kuadrant-system/authorino-757c8fdd8-w4knj"
Nov 02 01:40:05.137125 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.137085    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757c8fdd8-w4knj"
Nov 02 01:40:05.147942 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.147918    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrbrm"]
Nov 02 01:40:05.150517 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:40:05.150489    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefebec03_c7c7_4607_895b_fd7bb1b6a925.slice/crio-e5a165b765b62e0fac501cb718b0ee9c3e37c66b8a853837e3ffda63931ee856 WatchSource:0}: Error finding container e5a165b765b62e0fac501cb718b0ee9c3e37c66b8a853837e3ffda63931ee856: Status 404 returned error can't find the container with id e5a165b765b62e0fac501cb718b0ee9c3e37c66b8a853837e3ffda63931ee856
Nov 02 01:40:05.237435 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.237391    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm" event={"ID":"efebec03-c7c7-4607-895b-fd7bb1b6a925","Type":"ContainerStarted","Data":"e5a165b765b62e0fac501cb718b0ee9c3e37c66b8a853837e3ffda63931ee856"}
Nov 02 01:40:05.286852 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.286778    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-757c8fdd8-w4knj"]
Nov 02 01:40:05.318355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.318318    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7dcd7fd5c8-d75js"]
Nov 02 01:40:05.322445 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.322379    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:40:05.349851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.349815    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nm8v\" (UniqueName: \"kubernetes.io/projected/a25ed805-de3a-492b-8580-a0e95f693797-kube-api-access-8nm8v\") pod \"authorino-7dcd7fd5c8-d75js\" (UID: \"a25ed805-de3a-492b-8580-a0e95f693797\") " pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:40:05.350005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.349878    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a25ed805-de3a-492b-8580-a0e95f693797-tls-cert\") pod \"authorino-7dcd7fd5c8-d75js\" (UID: \"a25ed805-de3a-492b-8580-a0e95f693797\") " pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:40:05.375761 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.375679    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 01:40:05.378253 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.378219    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dcd7fd5c8-d75js"]
Nov 02 01:40:05.450579 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.450537    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a25ed805-de3a-492b-8580-a0e95f693797-tls-cert\") pod \"authorino-7dcd7fd5c8-d75js\" (UID: \"a25ed805-de3a-492b-8580-a0e95f693797\") " pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:40:05.450797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.450649    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nm8v\" (UniqueName: \"kubernetes.io/projected/a25ed805-de3a-492b-8580-a0e95f693797-kube-api-access-8nm8v\") pod \"authorino-7dcd7fd5c8-d75js\" (UID: \"a25ed805-de3a-492b-8580-a0e95f693797\") " pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:40:05.453250 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.453229    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a25ed805-de3a-492b-8580-a0e95f693797-tls-cert\") pod \"authorino-7dcd7fd5c8-d75js\" (UID: \"a25ed805-de3a-492b-8580-a0e95f693797\") " pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:40:05.512110 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.512076    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nm8v\" (UniqueName: \"kubernetes.io/projected/a25ed805-de3a-492b-8580-a0e95f693797-kube-api-access-8nm8v\") pod \"authorino-7dcd7fd5c8-d75js\" (UID: \"a25ed805-de3a-492b-8580-a0e95f693797\") " pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:40:05.517707 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.517563    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-757c8fdd8-w4knj"]
Nov 02 01:40:05.519861 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:40:05.519836    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52e40a04_8a80_476b_b34c_b4d18db25eb9.slice/crio-a583980feef9ea42925da34fa7cf84caddaa791980099196306bb373bf9520a9 WatchSource:0}: Error finding container a583980feef9ea42925da34fa7cf84caddaa791980099196306bb373bf9520a9: Status 404 returned error can't find the container with id a583980feef9ea42925da34fa7cf84caddaa791980099196306bb373bf9520a9
Nov 02 01:40:05.635168 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:05.635136    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:40:06.008952 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:06.008929    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dcd7fd5c8-d75js"]
Nov 02 01:40:06.010194 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:40:06.010169    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda25ed805_de3a_492b_8580_a0e95f693797.slice/crio-a9d0fa91020ad07b4c0223e67f49ec04afa0216af5a512f816ed526691022db8 WatchSource:0}: Error finding container a9d0fa91020ad07b4c0223e67f49ec04afa0216af5a512f816ed526691022db8: Status 404 returned error can't find the container with id a9d0fa91020ad07b4c0223e67f49ec04afa0216af5a512f816ed526691022db8
Nov 02 01:40:06.244021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:06.243986    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757c8fdd8-w4knj" event={"ID":"52e40a04-8a80-476b-b34c-b4d18db25eb9","Type":"ContainerStarted","Data":"a583980feef9ea42925da34fa7cf84caddaa791980099196306bb373bf9520a9"}
Nov 02 01:40:06.245359 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:06.245336    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js" event={"ID":"a25ed805-de3a-492b-8580-a0e95f693797","Type":"ContainerStarted","Data":"a9d0fa91020ad07b4c0223e67f49ec04afa0216af5a512f816ed526691022db8"}
Nov 02 01:40:07.253363 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:07.253302    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm" event={"ID":"efebec03-c7c7-4607-895b-fd7bb1b6a925","Type":"ContainerStarted","Data":"01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062"}
Nov 02 01:40:07.254925 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:07.253404    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm" podUID="efebec03-c7c7-4607-895b-fd7bb1b6a925" containerName="authorino" containerID="cri-o://01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062" gracePeriod=30
Nov 02 01:40:07.322109 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:07.322074    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nfdcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": read tcp 10.132.0.2:60858->10.132.0.32:15021: read: connection reset by peer" start-of-body=
Nov 02 01:40:07.322273 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:07.322138    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": read tcp 10.132.0.2:60858->10.132.0.32:15021: read: connection reset by peer"
Nov 02 01:40:07.367902 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:07.367843    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm" podStartSLOduration=1.529746831 podStartE2EDuration="3.367819601s" totalImagesPullingTime="1.83807277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:40:07.365754679 +0000 UTC m=+500.271220037" watchObservedRunningTime="2026-11-02 01:40:07.367819601 +0000 UTC m=+500.273284943"
Nov 02 01:40:07.513619 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:07.513561    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm"
Nov 02 01:40:07.571251 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:07.571229    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efebec03-c7c7-4607-895b-fd7bb1b6a925-kube-api-access-g2wkb\" (UniqueName: \"kubernetes.io/projected/efebec03-c7c7-4607-895b-fd7bb1b6a925-kube-api-access-g2wkb\") pod \"efebec03-c7c7-4607-895b-fd7bb1b6a925\" (UID: \"efebec03-c7c7-4607-895b-fd7bb1b6a925\") "
Nov 02 01:40:07.573254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:07.573228    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efebec03-c7c7-4607-895b-fd7bb1b6a925-kube-api-access-g2wkb" pod "efebec03-c7c7-4607-895b-fd7bb1b6a925" (UID: "efebec03-c7c7-4607-895b-fd7bb1b6a925"). InnerVolumeSpecName "kube-api-access-g2wkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:40:07.672443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:07.672415    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2wkb\" (UniqueName: \"kubernetes.io/projected/efebec03-c7c7-4607-895b-fd7bb1b6a925-kube-api-access-g2wkb\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:40:08.261335 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.261295    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757c8fdd8-w4knj" event={"ID":"52e40a04-8a80-476b-b34c-b4d18db25eb9","Type":"ContainerStarted","Data":"afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5"}
Nov 02 01:40:08.261853 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.261400    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-757c8fdd8-w4knj" podUID="52e40a04-8a80-476b-b34c-b4d18db25eb9" containerName="authorino" containerID="cri-o://afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5" gracePeriod=30
Nov 02 01:40:08.263041 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.263016    4912 generic.go:356] "Generic (PLEG): container finished" podID="efebec03-c7c7-4607-895b-fd7bb1b6a925" containerID="01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062" exitCode=0
Nov 02 01:40:08.263338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.263097    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm"
Nov 02 01:40:08.263338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.263105    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm" event={"ID":"efebec03-c7c7-4607-895b-fd7bb1b6a925","Type":"ContainerDied","Data":"01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062"}
Nov 02 01:40:08.263338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.263144    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lrbrm" event={"ID":"efebec03-c7c7-4607-895b-fd7bb1b6a925","Type":"ContainerDied","Data":"e5a165b765b62e0fac501cb718b0ee9c3e37c66b8a853837e3ffda63931ee856"}
Nov 02 01:40:08.263338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.263168    4912 scope.go:122] "RemoveContainer" containerID="01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062"
Nov 02 01:40:08.267984 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.267963    4912 generic.go:356] "Generic (PLEG): container finished" podID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerID="2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e" exitCode=0
Nov 02 01:40:08.268116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.268042    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" event={"ID":"80c31c29-3c02-42ed-b6ef-24518ff9fa4a","Type":"ContainerDied","Data":"2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e"}
Nov 02 01:40:08.268489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.268469    4912 scope.go:122] "RemoveContainer" containerID="2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e"
Nov 02 01:40:08.269753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.269732    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js" event={"ID":"a25ed805-de3a-492b-8580-a0e95f693797","Type":"ContainerStarted","Data":"f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8"}
Nov 02 01:40:08.269851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.269780    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:40:08.269899 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.269866    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:40:08.269935 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.269912    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:40:08.289518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.278483    4912 scope.go:122] "RemoveContainer" containerID="01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062"
Nov 02 01:40:08.289518 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:40:08.284369    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062\": container with ID starting with 01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062 not found: ID does not exist" containerID="01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062"
Nov 02 01:40:08.289518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.284407    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062"} err="failed to get container status \"01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062\": rpc error: code = NotFound desc = could not find container \"01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062\": container with ID starting with 01ac9e6388053f5b397f03553188ae0d71166bc42b1cfa7ade78625479e58062 not found: ID does not exist"
Nov 02 01:40:08.374607 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.374546    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-757c8fdd8-w4knj" podStartSLOduration=2.124320081 podStartE2EDuration="4.374527951s" totalImagesPullingTime="2.25020787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:40:08.373848903 +0000 UTC m=+501.279314261" watchObservedRunningTime="2026-11-02 01:40:08.374527951 +0000 UTC m=+501.279993317"
Nov 02 01:40:08.489958 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.488240    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrbrm"]
Nov 02 01:40:08.525805 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.525725    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrbrm"]
Nov 02 01:40:08.564815 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.564790    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757c8fdd8-w4knj"
Nov 02 01:40:08.681613 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.681578    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e40a04-8a80-476b-b34c-b4d18db25eb9-kube-api-access-w6p7w\" (UniqueName: \"kubernetes.io/projected/52e40a04-8a80-476b-b34c-b4d18db25eb9-kube-api-access-w6p7w\") pod \"52e40a04-8a80-476b-b34c-b4d18db25eb9\" (UID: \"52e40a04-8a80-476b-b34c-b4d18db25eb9\") "
Nov 02 01:40:08.684331 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.684299    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e40a04-8a80-476b-b34c-b4d18db25eb9-kube-api-access-w6p7w" pod "52e40a04-8a80-476b-b34c-b4d18db25eb9" (UID: "52e40a04-8a80-476b-b34c-b4d18db25eb9"). InnerVolumeSpecName "kube-api-access-w6p7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:40:08.725547 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.725470    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js" podStartSLOduration=2.179549535 podStartE2EDuration="3.725449967s" totalImagesPullingTime="1.545900432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:40:08.723927274 +0000 UTC m=+501.629392633" watchObservedRunningTime="2026-11-02 01:40:08.725449967 +0000 UTC m=+501.630915327"
Nov 02 01:40:08.782986 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:08.782904    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6p7w\" (UniqueName: \"kubernetes.io/projected/52e40a04-8a80-476b-b34c-b4d18db25eb9-kube-api-access-w6p7w\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:40:09.223950 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.223915    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efebec03-c7c7-4607-895b-fd7bb1b6a925" path="/var/lib/kubelet/pods/efebec03-c7c7-4607-895b-fd7bb1b6a925/volumes"
Nov 02 01:40:09.280802 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.280768    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" event={"ID":"80c31c29-3c02-42ed-b6ef-24518ff9fa4a","Type":"ContainerStarted","Data":"3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf"}
Nov 02 01:40:09.282282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.282256    4912 generic.go:356] "Generic (PLEG): container finished" podID="52e40a04-8a80-476b-b34c-b4d18db25eb9" containerID="afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5" exitCode=0
Nov 02 01:40:09.282282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.282283    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757c8fdd8-w4knj"
Nov 02 01:40:09.282484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.282290    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757c8fdd8-w4knj" event={"ID":"52e40a04-8a80-476b-b34c-b4d18db25eb9","Type":"ContainerDied","Data":"afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5"}
Nov 02 01:40:09.282484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.282331    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757c8fdd8-w4knj" event={"ID":"52e40a04-8a80-476b-b34c-b4d18db25eb9","Type":"ContainerDied","Data":"a583980feef9ea42925da34fa7cf84caddaa791980099196306bb373bf9520a9"}
Nov 02 01:40:09.282484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.282347    4912 scope.go:122] "RemoveContainer" containerID="afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5"
Nov 02 01:40:09.291723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.291705    4912 scope.go:122] "RemoveContainer" containerID="afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5"
Nov 02 01:40:09.291965 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:40:09.291948    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5\": container with ID starting with afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5 not found: ID does not exist" containerID="afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5"
Nov 02 01:40:09.292029 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.291972    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5"} err="failed to get container status \"afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5\": rpc error: code = NotFound desc = could not find container \"afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5\": container with ID starting with afda2ddace3a82695c19fc057a63b8b2a9e5fc3db3a186a1adc259b27dc9c5c5 not found: ID does not exist"
Nov 02 01:40:09.571392 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.571247    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-757c8fdd8-w4knj"]
Nov 02 01:40:09.576089 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.576036    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-757c8fdd8-w4knj"]
Nov 02 01:40:09.907570 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.907529    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:40:09.907770 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.907704    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 01:40:09.907821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:09.907768    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 01:40:10.094752 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:10.094722    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:40:10.097839 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:10.097812    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dpzkp"
Nov 02 01:40:10.907564 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:10.907527    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 01:40:10.908112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:10.907597    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 01:40:11.224911 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:11.224843    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e40a04-8a80-476b-b34c-b4d18db25eb9" path="/var/lib/kubelet/pods/52e40a04-8a80-476b-b34c-b4d18db25eb9/volumes"
Nov 02 01:40:11.907380 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:11.907342    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 01:40:11.907554 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:11.907405    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 01:40:12.138279 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:40:12.138221    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:40:12.907557 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:12.907522    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 01:40:12.907740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:12.907589    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 01:40:13.907208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:13.907174    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 01:40:13.907619 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:13.907238    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 01:40:14.911006 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:14.910975    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:40:14.911431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:14.911303    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:40:14.911895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:14.911876    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:40:40.462437 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:40:40.462355    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:40:42.136845 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:40:42.136808    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:41:12.136917 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:41:12.136878    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:41:23.364211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:23.364176    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:41:23.538953 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:23.538917    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:41:24.984724 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:24.984685    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"]
Nov 02 01:41:24.985257 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:24.984930    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx" podUID="1ac4161e-a1fc-40e4-8e87-46b77007d077" containerName="manager" containerID="cri-o://e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003" gracePeriod=30
Nov 02 01:41:25.092429 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.092384    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx" podUID="1ac4161e-a1fc-40e4-8e87-46b77007d077" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.52:8081/readyz\": dial tcp 10.132.0.52:8081: connect: connection refused"
Nov 02 01:41:25.249021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.248960    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:41:25.360337 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.360297    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac4161e-a1fc-40e4-8e87-46b77007d077-kube-api-access-9vtvb\" (UniqueName: \"kubernetes.io/projected/1ac4161e-a1fc-40e4-8e87-46b77007d077-kube-api-access-9vtvb\") pod \"1ac4161e-a1fc-40e4-8e87-46b77007d077\" (UID: \"1ac4161e-a1fc-40e4-8e87-46b77007d077\") "
Nov 02 01:41:25.360528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.360395    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ac4161e-a1fc-40e4-8e87-46b77007d077-cert\" (UniqueName: \"kubernetes.io/secret/1ac4161e-a1fc-40e4-8e87-46b77007d077-cert\") pod \"1ac4161e-a1fc-40e4-8e87-46b77007d077\" (UID: \"1ac4161e-a1fc-40e4-8e87-46b77007d077\") "
Nov 02 01:41:25.362672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.362645    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac4161e-a1fc-40e4-8e87-46b77007d077-kube-api-access-9vtvb" pod "1ac4161e-a1fc-40e4-8e87-46b77007d077" (UID: "1ac4161e-a1fc-40e4-8e87-46b77007d077"). InnerVolumeSpecName "kube-api-access-9vtvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:41:25.362789 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.362770    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac4161e-a1fc-40e4-8e87-46b77007d077-cert" pod "1ac4161e-a1fc-40e4-8e87-46b77007d077" (UID: "1ac4161e-a1fc-40e4-8e87-46b77007d077"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:41:25.461967 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.461928    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vtvb\" (UniqueName: \"kubernetes.io/projected/1ac4161e-a1fc-40e4-8e87-46b77007d077-kube-api-access-9vtvb\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:25.461967 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.461963    4912 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ac4161e-a1fc-40e4-8e87-46b77007d077-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:41:25.695351 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.695314    4912 generic.go:356] "Generic (PLEG): container finished" podID="1ac4161e-a1fc-40e4-8e87-46b77007d077" containerID="e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003" exitCode=0
Nov 02 01:41:25.695541 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.695362    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"
Nov 02 01:41:25.695541 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.695395    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx" event={"ID":"1ac4161e-a1fc-40e4-8e87-46b77007d077","Type":"ContainerDied","Data":"e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003"}
Nov 02 01:41:25.695541 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.695445    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx" event={"ID":"1ac4161e-a1fc-40e4-8e87-46b77007d077","Type":"ContainerDied","Data":"6371be06397a0b2cd3b0f3aac56b3c0511ef088a4669be9455e49fd36056982c"}
Nov 02 01:41:25.695541 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.695467    4912 scope.go:122] "RemoveContainer" containerID="e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003"
Nov 02 01:41:25.705159 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.705137    4912 scope.go:122] "RemoveContainer" containerID="e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003"
Nov 02 01:41:25.705479 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:41:25.705461    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003\": container with ID starting with e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003 not found: ID does not exist" containerID="e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003"
Nov 02 01:41:25.705536 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.705488    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003"} err="failed to get container status \"e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003\": rpc error: code = NotFound desc = could not find container \"e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003\": container with ID starting with e19ef0ab0a26a98dbe982d3d09cd8fb9cfdd8261d7fc8901722d9a3a4f350003 not found: ID does not exist"
Nov 02 01:41:25.830418 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.830385    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:41:25.868812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.868781    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"]
Nov 02 01:41:25.918340 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:25.918308    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5vwsx"]
Nov 02 01:41:26.004308 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:26.004220    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:41:27.224351 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:27.224275    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac4161e-a1fc-40e4-8e87-46b77007d077" path="/var/lib/kubelet/pods/1ac4161e-a1fc-40e4-8e87-46b77007d077/volumes"
Nov 02 01:41:40.792845 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:40.792799    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:41:42.138407 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:41:42.138366    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:41:46.448737 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:46.448705    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:41:47.417200 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:47.417171    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:41:47.423740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:47.423714    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:41:47.461074 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:47.461041    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:41:47.467837 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:47.467813    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:41:50.075518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.075480    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"]
Nov 02 01:41:50.076205 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.076185    4912 cpu_manager.go:414] "containerMap: removing container" podUID="efebec03-c7c7-4607-895b-fd7bb1b6a925" containerName="authorino"
Nov 02 01:41:50.076314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.076209    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efebec03-c7c7-4607-895b-fd7bb1b6a925" containerName="authorino"
Nov 02 01:41:50.076314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.076220    4912 cpu_manager.go:414] "containerMap: removing container" podUID="52e40a04-8a80-476b-b34c-b4d18db25eb9" containerName="authorino"
Nov 02 01:41:50.076314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.076228    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e40a04-8a80-476b-b34c-b4d18db25eb9" containerName="authorino"
Nov 02 01:41:50.076314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.076266    4912 cpu_manager.go:414] "containerMap: removing container" podUID="1ac4161e-a1fc-40e4-8e87-46b77007d077" containerName="manager"
Nov 02 01:41:50.076314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.076274    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac4161e-a1fc-40e4-8e87-46b77007d077" containerName="manager"
Nov 02 01:41:50.076565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.076378    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac4161e-a1fc-40e4-8e87-46b77007d077" containerName="manager"
Nov 02 01:41:50.076565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.076399    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="efebec03-c7c7-4607-895b-fd7bb1b6a925" containerName="authorino"
Nov 02 01:41:50.076565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.076410    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e40a04-8a80-476b-b34c-b4d18db25eb9" containerName="authorino"
Nov 02 01:41:50.080945 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.080919    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.148393 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.148347    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 01:41:50.184892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.184850    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 01:41:50.185151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.185128    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7vwh8\""
Nov 02 01:41:50.185355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.185333    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 01:41:50.195650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.195610    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"]
Nov 02 01:41:50.196140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.196103    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.196274 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.196155    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.196274 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.196244    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfqhb\" (UniqueName: \"kubernetes.io/projected/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-kube-api-access-nfqhb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.196448 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.196408    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.196711 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.196693    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.197265 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.197237    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.197770 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.197734    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:41:50.298471 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.298407    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.298471 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.298475    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.298776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.298510    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfqhb\" (UniqueName: \"kubernetes.io/projected/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-kube-api-access-nfqhb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.298776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.298538    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.298776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.298564    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.298776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.298679    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.299004 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.298954    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.299186 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.299162    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.299560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.299536    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.302615 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.302364    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.302615 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.302573    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.359945 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.359866    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfqhb\" (UniqueName: \"kubernetes.io/projected/62cc99ac-a5bc-4c99-a2e6-8487bb9f994e-kube-api-access-nfqhb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg\" (UID: \"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.398521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.398233    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:50.801319 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.801293    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"]
Nov 02 01:41:50.803699 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:41:50.803670    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cc99ac_a5bc_4c99_a2e6_8487bb9f994e.slice/crio-3ce4c5df2b5eb4cdf29cf6a8851c5912d76496a04c8d0da41f235c2eb8add92a WatchSource:0}: Error finding container 3ce4c5df2b5eb4cdf29cf6a8851c5912d76496a04c8d0da41f235c2eb8add92a: Status 404 returned error can't find the container with id 3ce4c5df2b5eb4cdf29cf6a8851c5912d76496a04c8d0da41f235c2eb8add92a
Nov 02 01:41:50.829679 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:50.829642    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg" event={"ID":"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e","Type":"ContainerStarted","Data":"3ce4c5df2b5eb4cdf29cf6a8851c5912d76496a04c8d0da41f235c2eb8add92a"}
Nov 02 01:41:52.845358 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:52.845318    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg" event={"ID":"62cc99ac-a5bc-4c99-a2e6-8487bb9f994e","Type":"ContainerStarted","Data":"4683127817c1e74baf03c60e6a0c779ab65147956c1276a33d5a69ef7af35780"}
Nov 02 01:41:52.975710 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:52.975610    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg" podStartSLOduration=1.472961387 podStartE2EDuration="2.975560633s" totalImagesPullingTime="1.502599246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:52.970326133 +0000 UTC m=+605.875791490" watchObservedRunningTime="2026-11-02 01:41:52.975560633 +0000 UTC m=+605.881025990"
Nov 02 01:41:53.399089 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:53.399036    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:41:57.010450 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.010410    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"]
Nov 02 01:41:57.014463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.014437    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.067210 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.067181    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 01:41:57.067369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.067301    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94358ff5-ab4d-4e82-880f-515d87a4a790-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.067369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.067361    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.067482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.067461    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.067521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.067492    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.067521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.067509    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nz96\" (UniqueName: \"kubernetes.io/projected/94358ff5-ab4d-4e82-880f-515d87a4a790-kube-api-access-5nz96\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.067627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.067609    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.118567 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.118527    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"]
Nov 02 01:41:57.139245 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.139201    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:41:57.169067 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.169020    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.169213 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.169119    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94358ff5-ab4d-4e82-880f-515d87a4a790-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.169213 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.169177    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.169317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.169231    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.169392 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.169369    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.169453 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.169415    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nz96\" (UniqueName: \"kubernetes.io/projected/94358ff5-ab4d-4e82-880f-515d87a4a790-kube-api-access-5nz96\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.169453 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.169433    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.169557 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.169521    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.169625 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.169552    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.171648 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.171627    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94358ff5-ab4d-4e82-880f-515d87a4a790-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.171784 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.171769    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94358ff5-ab4d-4e82-880f-515d87a4a790-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.230377 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.230344    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nz96\" (UniqueName: \"kubernetes.io/projected/94358ff5-ab4d-4e82-880f-515d87a4a790-kube-api-access-5nz96\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p\" (UID: \"94358ff5-ab4d-4e82-880f-515d87a4a790\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.327846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.327760    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:41:57.706574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.706550    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"]
Nov 02 01:41:57.707701 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:41:57.707674    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94358ff5_ab4d_4e82_880f_515d87a4a790.slice/crio-2dd2081d45907a7a5452ea00ff9ea44b823a5a437461d17203117e94ed0a21a7 WatchSource:0}: Error finding container 2dd2081d45907a7a5452ea00ff9ea44b823a5a437461d17203117e94ed0a21a7: Status 404 returned error can't find the container with id 2dd2081d45907a7a5452ea00ff9ea44b823a5a437461d17203117e94ed0a21a7
Nov 02 01:41:57.708983 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.708965    4912 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:41:57.877221 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:57.877188    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p" event={"ID":"94358ff5-ab4d-4e82-880f-515d87a4a790","Type":"ContainerStarted","Data":"2dd2081d45907a7a5452ea00ff9ea44b823a5a437461d17203117e94ed0a21a7"}
Nov 02 01:41:58.884281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:58.884246    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p" event={"ID":"94358ff5-ab4d-4e82-880f-515d87a4a790","Type":"ContainerStarted","Data":"153470c390540e16f18c1beb2bdb43667506be11e69097f7e968204fe98f76a7"}
Nov 02 01:41:59.002222 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:59.002173    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p" podStartSLOduration=2.399790782 podStartE2EDuration="3.002158242s" totalImagesPullingTime="602.36746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:59.000979592 +0000 UTC m=+611.906444979" watchObservedRunningTime="2026-11-02 01:41:59.002158242 +0000 UTC m=+611.907623598"
Nov 02 01:41:59.328179 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:41:59.328102    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:42:00.399352 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:00.399324    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:42:00.416143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:00.416121    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:42:00.922957 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:00.922925    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg"
Nov 02 01:42:01.136881 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:01.136846    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:04.735652 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.735615    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"]
Nov 02 01:42:04.739337 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.739317    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.842136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.842099    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.842339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.842310    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825d5fda-b233-458a-93f7-31f99cecadb3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.842400 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.842374    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.842454 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.842402    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.842523 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.842481    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.842563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.842537    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brj9p\" (UniqueName: \"kubernetes.io/projected/825d5fda-b233-458a-93f7-31f99cecadb3-kube-api-access-brj9p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.842629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.842609    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 01:42:04.845197 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.845172    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"]
Nov 02 01:42:04.858353 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.858325    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:04.943315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.943284    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.943525 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.943365    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825d5fda-b233-458a-93f7-31f99cecadb3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.943525 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.943389    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.943525 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.943409    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.943525 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.943435    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.943525 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.943457    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brj9p\" (UniqueName: \"kubernetes.io/projected/825d5fda-b233-458a-93f7-31f99cecadb3-kube-api-access-brj9p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.943815 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.943790    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.943890 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.943866    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.943976 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.943949    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.946095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.946070    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825d5fda-b233-458a-93f7-31f99cecadb3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:04.946431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:04.946411    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825d5fda-b233-458a-93f7-31f99cecadb3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:05.004550 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:05.004471    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brj9p\" (UniqueName: \"kubernetes.io/projected/825d5fda-b233-458a-93f7-31f99cecadb3-kube-api-access-brj9p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pgx5n\" (UID: \"825d5fda-b233-458a-93f7-31f99cecadb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:05.052439 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:05.052405    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:05.430585 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:05.430554    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"]
Nov 02 01:42:05.434271 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:42:05.434231    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825d5fda_b233_458a_93f7_31f99cecadb3.slice/crio-a7fe497291bc086d3087f3739a462b37b2ef9000a923a2195faeb1214e344ab2 WatchSource:0}: Error finding container a7fe497291bc086d3087f3739a462b37b2ef9000a923a2195faeb1214e344ab2: Status 404 returned error can't find the container with id a7fe497291bc086d3087f3739a462b37b2ef9000a923a2195faeb1214e344ab2
Nov 02 01:42:05.933009 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:05.932975    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n" event={"ID":"825d5fda-b233-458a-93f7-31f99cecadb3","Type":"ContainerStarted","Data":"a7fe497291bc086d3087f3739a462b37b2ef9000a923a2195faeb1214e344ab2"}
Nov 02 01:42:06.941121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:06.941052    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n" event={"ID":"825d5fda-b233-458a-93f7-31f99cecadb3","Type":"ContainerStarted","Data":"86213fdb5c3591a2dbd9baffba555d69493398ff44f64969ea4f4a7b249f89e8"}
Nov 02 01:42:07.053045 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:07.053017    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:07.061871 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:07.061815    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n" podStartSLOduration=2.474927377 podStartE2EDuration="3.061799163s" totalImagesPullingTime="586.871786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:07.058241192 +0000 UTC m=+619.963706548" watchObservedRunningTime="2026-11-02 01:42:07.061799163 +0000 UTC m=+619.967264522"
Nov 02 01:42:07.328350 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:07.328265    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:42:07.340955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:07.340925    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:42:07.958738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:07.958710    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p"
Nov 02 01:42:08.161538 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:08.161505    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:11.649036 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:11.648997    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:12.141672 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:42:12.141636    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:42:15.052555 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.052521    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:15.067838 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.067810    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:15.293253 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.293210    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-46j9s"]
Nov 02 01:42:15.302418 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.302396    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.353521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.353489    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-46j9s"]
Nov 02 01:42:15.354503 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.354484    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 01:42:15.442456 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.442418    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5d7\" (UniqueName: \"kubernetes.io/projected/8df16c8c-28c7-46e5-a791-0bed28ecb05a-kube-api-access-bk5d7\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.442616 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.442493    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-home\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.442616 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.442588    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.442697 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.442613    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.442697 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.442645    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.442765 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.442701    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df16c8c-28c7-46e5-a791-0bed28ecb05a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.461486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.461454    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:15.544192 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.544157    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df16c8c-28c7-46e5-a791-0bed28ecb05a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.544391 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.544229    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5d7\" (UniqueName: \"kubernetes.io/projected/8df16c8c-28c7-46e5-a791-0bed28ecb05a-kube-api-access-bk5d7\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.544391 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.544304    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-home\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.544391 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.544347    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.544391 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.544370    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.544593 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.544402    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.544817 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.544780    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-home\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.544924 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.544813    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.544924 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.544828    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.546737 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.546714    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df16c8c-28c7-46e5-a791-0bed28ecb05a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.546975 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.546958    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df16c8c-28c7-46e5-a791-0bed28ecb05a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.605235 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.605165    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5d7\" (UniqueName: \"kubernetes.io/projected/8df16c8c-28c7-46e5-a791-0bed28ecb05a-kube-api-access-bk5d7\") pod \"echo-simulator-kserve-64b6f8f96c-46j9s\" (UID: \"8df16c8c-28c7-46e5-a791-0bed28ecb05a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:15.618116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:15.618088    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:16.000245 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:16.000222    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-46j9s"]
Nov 02 01:42:16.002033 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:42:16.002008    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df16c8c_28c7_46e5_a791_0bed28ecb05a.slice/crio-e2b0f2ae60fd98a26b89ad782c55ead5ca1813312e3fd745f1a101870cb80756 WatchSource:0}: Error finding container e2b0f2ae60fd98a26b89ad782c55ead5ca1813312e3fd745f1a101870cb80756: Status 404 returned error can't find the container with id e2b0f2ae60fd98a26b89ad782c55ead5ca1813312e3fd745f1a101870cb80756
Nov 02 01:42:16.005909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:16.005887    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n"
Nov 02 01:42:17.001992 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:17.001950    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s" event={"ID":"8df16c8c-28c7-46e5-a791-0bed28ecb05a","Type":"ContainerStarted","Data":"e2b0f2ae60fd98a26b89ad782c55ead5ca1813312e3fd745f1a101870cb80756"}
Nov 02 01:42:19.019014 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:19.018893    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s" event={"ID":"8df16c8c-28c7-46e5-a791-0bed28ecb05a","Type":"ContainerStarted","Data":"54ba5529d39d48a35b833f8dbc62ab51267e232348b4428c46f0eed38a8bb225"}
Nov 02 01:42:19.139246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:19.139196    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s" podStartSLOduration=1.520804773 podStartE2EDuration="4.139180844s" totalImagesPullingTime="2.618376071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:19.138318343 +0000 UTC m=+632.043783699" watchObservedRunningTime="2026-11-02 01:42:19.139180844 +0000 UTC m=+632.044646202"
Nov 02 01:42:19.619068 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:19.619024    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:24.067017 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.066984    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"]
Nov 02 01:42:24.072943 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.072924    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.239697 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.239664    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br7hk\" (UniqueName: \"kubernetes.io/projected/ea71fe46-235c-43bb-adc6-708430a55abb-kube-api-access-br7hk\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.239849 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.239729    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.239909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.239854    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.239909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.239878    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.239981 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.239908    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.240098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.240044    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.240174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.240112    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea71fe46-235c-43bb-adc6-708430a55abb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.240174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.240139    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea71fe46-235c-43bb-adc6-708430a55abb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.240174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.240164    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea71fe46-235c-43bb-adc6-708430a55abb-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.271613 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.271582    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"]
Nov 02 01:42:24.341108 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.340976    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341108 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341033    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341108 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341083    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341469 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341131    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341469 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341169    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341469 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341198    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea71fe46-235c-43bb-adc6-708430a55abb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341469 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341234    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea71fe46-235c-43bb-adc6-708430a55abb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341469 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341273    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea71fe46-235c-43bb-adc6-708430a55abb-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341469 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341319    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7hk\" (UniqueName: \"kubernetes.io/projected/ea71fe46-235c-43bb-adc6-708430a55abb-kube-api-access-br7hk\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341469 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341463    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341615    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341646    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.341846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.341671    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.342049 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.342031    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea71fe46-235c-43bb-adc6-708430a55abb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.343800 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.343778    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea71fe46-235c-43bb-adc6-708430a55abb-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.344190 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.344171    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea71fe46-235c-43bb-adc6-708430a55abb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.402021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.401987    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea71fe46-235c-43bb-adc6-708430a55abb-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.402197 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.402078    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7hk\" (UniqueName: \"kubernetes.io/projected/ea71fe46-235c-43bb-adc6-708430a55abb-kube-api-access-br7hk\") pod \"maas-default-gateway-openshift-default-778b7b5c47-4tqpd\" (UID: \"ea71fe46-235c-43bb-adc6-708430a55abb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:24.686708 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:24.686673    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:25.117701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:25.117679    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"]
Nov 02 01:42:25.119525 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:42:25.119503    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea71fe46_235c_43bb_adc6_708430a55abb.slice/crio-299ec124c4a8546bde7e1715fae6c7cbc6c76a22fb54720a6eb954bab6f3dea6 WatchSource:0}: Error finding container 299ec124c4a8546bde7e1715fae6c7cbc6c76a22fb54720a6eb954bab6f3dea6: Status 404 returned error can't find the container with id 299ec124c4a8546bde7e1715fae6c7cbc6c76a22fb54720a6eb954bab6f3dea6
Nov 02 01:42:25.121643 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:25.121606    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:42:25.121725 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:25.121675    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:42:25.121725 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:25.121707    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:42:25.618551 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:25.618524    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:25.631131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:25.631105    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:26.087241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:26.087143    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" event={"ID":"ea71fe46-235c-43bb-adc6-708430a55abb","Type":"ContainerStarted","Data":"06d445068330f07296efacff92431460dd60e9e38c06135b23c877942939b14d"}
Nov 02 01:42:26.087241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:26.087214    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" event={"ID":"ea71fe46-235c-43bb-adc6-708430a55abb","Type":"ContainerStarted","Data":"299ec124c4a8546bde7e1715fae6c7cbc6c76a22fb54720a6eb954bab6f3dea6"}
Nov 02 01:42:26.098919 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:26.098892    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s"
Nov 02 01:42:26.207796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:26.207741    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podStartSLOduration=2.207725832 podStartE2EDuration="2.207725832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:26.206579722 +0000 UTC m=+639.112045070" watchObservedRunningTime="2026-11-02 01:42:26.207725832 +0000 UTC m=+639.113191188"
Nov 02 01:42:26.687497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:26.687469    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:26.688968 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:26.688934    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4tqpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:42:26.689127 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:26.688988    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podUID="ea71fe46-235c-43bb-adc6-708430a55abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:42:27.687654 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:27.687619    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4tqpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:42:27.688085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:27.687675    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podUID="ea71fe46-235c-43bb-adc6-708430a55abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:42:28.687696 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:28.687658    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4tqpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:42:28.688129 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:28.687716    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podUID="ea71fe46-235c-43bb-adc6-708430a55abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:42:29.687184 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:29.687133    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4tqpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:42:29.687404 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:29.687211    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podUID="ea71fe46-235c-43bb-adc6-708430a55abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:42:30.687709 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:30.687668    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4tqpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:42:30.688128 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:30.687735    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podUID="ea71fe46-235c-43bb-adc6-708430a55abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:42:31.687321 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:31.687280    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4tqpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:42:31.687511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:31.687365    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podUID="ea71fe46-235c-43bb-adc6-708430a55abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:42:33.368180 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:33.368149    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:33.368658 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:33.368553    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:33.369402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:33.369380    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd"
Nov 02 01:42:40.459012 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.458923    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76fddf7-gwxbn"]
Nov 02 01:42:40.463386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.463363    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76fddf7-gwxbn"
Nov 02 01:42:40.515494 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.515458    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9t56\" (UniqueName: \"kubernetes.io/projected/d7f049e4-ddcb-45e1-8819-2919806f1f3b-kube-api-access-h9t56\") pod \"authorino-6dd76fddf7-gwxbn\" (UID: \"d7f049e4-ddcb-45e1-8819-2919806f1f3b\") " pod="kuadrant-system/authorino-6dd76fddf7-gwxbn"
Nov 02 01:42:40.515696 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.515632    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7f049e4-ddcb-45e1-8819-2919806f1f3b-tls-cert\") pod \"authorino-6dd76fddf7-gwxbn\" (UID: \"d7f049e4-ddcb-45e1-8819-2919806f1f3b\") " pod="kuadrant-system/authorino-6dd76fddf7-gwxbn"
Nov 02 01:42:40.519266 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.519239    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76fddf7-gwxbn"]
Nov 02 01:42:40.616655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.616620    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9t56\" (UniqueName: \"kubernetes.io/projected/d7f049e4-ddcb-45e1-8819-2919806f1f3b-kube-api-access-h9t56\") pod \"authorino-6dd76fddf7-gwxbn\" (UID: \"d7f049e4-ddcb-45e1-8819-2919806f1f3b\") " pod="kuadrant-system/authorino-6dd76fddf7-gwxbn"
Nov 02 01:42:40.616840 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.616720    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7f049e4-ddcb-45e1-8819-2919806f1f3b-tls-cert\") pod \"authorino-6dd76fddf7-gwxbn\" (UID: \"d7f049e4-ddcb-45e1-8819-2919806f1f3b\") " pod="kuadrant-system/authorino-6dd76fddf7-gwxbn"
Nov 02 01:42:40.619299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.619276    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7f049e4-ddcb-45e1-8819-2919806f1f3b-tls-cert\") pod \"authorino-6dd76fddf7-gwxbn\" (UID: \"d7f049e4-ddcb-45e1-8819-2919806f1f3b\") " pod="kuadrant-system/authorino-6dd76fddf7-gwxbn"
Nov 02 01:42:40.677077 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.677032    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9t56\" (UniqueName: \"kubernetes.io/projected/d7f049e4-ddcb-45e1-8819-2919806f1f3b-kube-api-access-h9t56\") pod \"authorino-6dd76fddf7-gwxbn\" (UID: \"d7f049e4-ddcb-45e1-8819-2919806f1f3b\") " pod="kuadrant-system/authorino-6dd76fddf7-gwxbn"
Nov 02 01:42:40.681749 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.681704    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:40.776320 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.776243    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76fddf7-gwxbn"
Nov 02 01:42:40.907942 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:40.907905    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:41.141272 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:41.141233    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:41.150205 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:41.150182    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76fddf7-gwxbn"]
Nov 02 01:42:41.151849 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:42:41.151824    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7f049e4_ddcb_45e1_8819_2919806f1f3b.slice/crio-05ec01bed78654cf51f182c06389b7cc2f9b0b0df13b066b746a99c44d2bab8e WatchSource:0}: Error finding container 05ec01bed78654cf51f182c06389b7cc2f9b0b0df13b066b746a99c44d2bab8e: Status 404 returned error can't find the container with id 05ec01bed78654cf51f182c06389b7cc2f9b0b0df13b066b746a99c44d2bab8e
Nov 02 01:42:41.183197 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:41.183171    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd76fddf7-gwxbn" event={"ID":"d7f049e4-ddcb-45e1-8819-2919806f1f3b","Type":"ContainerStarted","Data":"05ec01bed78654cf51f182c06389b7cc2f9b0b0df13b066b746a99c44d2bab8e"}
Nov 02 01:42:42.136874 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:42:42.136828    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:42:43.195698 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.195665    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd76fddf7-gwxbn" event={"ID":"d7f049e4-ddcb-45e1-8819-2919806f1f3b","Type":"ContainerStarted","Data":"df80f6b7e423403751759056ae065e3441549f557b1297be32017c79347eb217"}
Nov 02 01:42:43.311771 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.311721    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6dd76fddf7-gwxbn" podStartSLOduration=1.881388749 podStartE2EDuration="3.31170569s" totalImagesPullingTime="1.430316941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:43.309325202 +0000 UTC m=+656.214790560" watchObservedRunningTime="2026-11-02 01:42:43.31170569 +0000 UTC m=+656.217171047"
Nov 02 01:42:43.362439 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.362201    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dcd7fd5c8-d75js"]
Nov 02 01:42:43.362439 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.362420    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js" podUID="a25ed805-de3a-492b-8580-a0e95f693797" containerName="authorino" containerID="cri-o://f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8" gracePeriod=30
Nov 02 01:42:43.641992 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.641965    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:42:43.747238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.747153    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25ed805-de3a-492b-8580-a0e95f693797-tls-cert\" (UniqueName: \"kubernetes.io/secret/a25ed805-de3a-492b-8580-a0e95f693797-tls-cert\") pod \"a25ed805-de3a-492b-8580-a0e95f693797\" (UID: \"a25ed805-de3a-492b-8580-a0e95f693797\") "
Nov 02 01:42:43.747238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.747224    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25ed805-de3a-492b-8580-a0e95f693797-kube-api-access-8nm8v\" (UniqueName: \"kubernetes.io/projected/a25ed805-de3a-492b-8580-a0e95f693797-kube-api-access-8nm8v\") pod \"a25ed805-de3a-492b-8580-a0e95f693797\" (UID: \"a25ed805-de3a-492b-8580-a0e95f693797\") "
Nov 02 01:42:43.749831 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.749800    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25ed805-de3a-492b-8580-a0e95f693797-kube-api-access-8nm8v" pod "a25ed805-de3a-492b-8580-a0e95f693797" (UID: "a25ed805-de3a-492b-8580-a0e95f693797"). InnerVolumeSpecName "kube-api-access-8nm8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:43.760790 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.760761    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25ed805-de3a-492b-8580-a0e95f693797-tls-cert" pod "a25ed805-de3a-492b-8580-a0e95f693797" (UID: "a25ed805-de3a-492b-8580-a0e95f693797"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:43.848878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.848845    4912 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a25ed805-de3a-492b-8580-a0e95f693797-tls-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:43.849158 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:43.849134    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nm8v\" (UniqueName: \"kubernetes.io/projected/a25ed805-de3a-492b-8580-a0e95f693797-kube-api-access-8nm8v\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:42:44.204158 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:44.204121    4912 generic.go:356] "Generic (PLEG): container finished" podID="a25ed805-de3a-492b-8580-a0e95f693797" containerID="f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8" exitCode=0
Nov 02 01:42:44.204627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:44.204166    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js"
Nov 02 01:42:44.204627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:44.204212    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js" event={"ID":"a25ed805-de3a-492b-8580-a0e95f693797","Type":"ContainerDied","Data":"f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8"}
Nov 02 01:42:44.204627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:44.204263    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dcd7fd5c8-d75js" event={"ID":"a25ed805-de3a-492b-8580-a0e95f693797","Type":"ContainerDied","Data":"a9d0fa91020ad07b4c0223e67f49ec04afa0216af5a512f816ed526691022db8"}
Nov 02 01:42:44.204627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:44.204281    4912 scope.go:122] "RemoveContainer" containerID="f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8"
Nov 02 01:42:44.216853 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:44.216832    4912 scope.go:122] "RemoveContainer" containerID="f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8"
Nov 02 01:42:44.217175 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:42:44.217155    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8\": container with ID starting with f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8 not found: ID does not exist" containerID="f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8"
Nov 02 01:42:44.217249 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:44.217184    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8"} err="failed to get container status \"f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8\": rpc error: code = NotFound desc = could not find container \"f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8\": container with ID starting with f9f594b73ab06985c59c0e2329f6cb6de9034b1609ba1ecf9f8d5625a7f6b2f8 not found: ID does not exist"
Nov 02 01:42:44.372036 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:44.372006    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dcd7fd5c8-d75js"]
Nov 02 01:42:44.376238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:44.376212    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7dcd7fd5c8-d75js"]
Nov 02 01:42:45.225035 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:45.224999    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25ed805-de3a-492b-8580-a0e95f693797" path="/var/lib/kubelet/pods/a25ed805-de3a-492b-8580-a0e95f693797/volumes"
Nov 02 01:42:45.286876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:45.286845    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:45.524131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:45.524029    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:51.552956 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:51.552920    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:51.874983 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:51.874941    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:42:51.992206 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:51.992176    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"]
Nov 02 01:42:51.992648 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:51.992635    4912 cpu_manager.go:414] "containerMap: removing container" podUID="a25ed805-de3a-492b-8580-a0e95f693797" containerName="authorino"
Nov 02 01:42:51.992692 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:51.992650    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25ed805-de3a-492b-8580-a0e95f693797" containerName="authorino"
Nov 02 01:42:51.992736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:51.992727    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25ed805-de3a-492b-8580-a0e95f693797" containerName="authorino"
Nov 02 01:42:52.000585 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.000559    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:42:52.056104 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.056046    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"]
Nov 02 01:42:52.101099 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.101047    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zk5rs\""
Nov 02 01:42:52.101341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.101092    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 01:42:52.110137 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.109986    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"]
Nov 02 01:42:52.114671 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.114648    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:42:52.129131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.129042    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjgjj\" (UniqueName: \"kubernetes.io/projected/a7d724fe-3d4a-493e-9783-3ad4f3d0f500-kube-api-access-pjgjj\") pod \"payload-pre-processing-7b98bb678c-dtmjd\" (UID: \"a7d724fe-3d4a-493e-9783-3ad4f3d0f500\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:42:52.129251 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.129205    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d724fe-3d4a-493e-9783-3ad4f3d0f500-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dtmjd\" (UID: \"a7d724fe-3d4a-493e-9783-3ad4f3d0f500\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:42:52.171442 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.171408    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"]
Nov 02 01:42:52.230560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.230518    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bd54dd-f5ec-4a45-b1fc-8534899e9b64-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fv5bz\" (UID: \"a2bd54dd-f5ec-4a45-b1fc-8534899e9b64\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:42:52.230754 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.230563    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjgjj\" (UniqueName: \"kubernetes.io/projected/a7d724fe-3d4a-493e-9783-3ad4f3d0f500-kube-api-access-pjgjj\") pod \"payload-pre-processing-7b98bb678c-dtmjd\" (UID: \"a7d724fe-3d4a-493e-9783-3ad4f3d0f500\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:42:52.230754 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.230657    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mrsr\" (UniqueName: \"kubernetes.io/projected/a2bd54dd-f5ec-4a45-b1fc-8534899e9b64-kube-api-access-6mrsr\") pod \"payload-processing-55fbbbf9dc-fv5bz\" (UID: \"a2bd54dd-f5ec-4a45-b1fc-8534899e9b64\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:42:52.230754 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.230693    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d724fe-3d4a-493e-9783-3ad4f3d0f500-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dtmjd\" (UID: \"a7d724fe-3d4a-493e-9783-3ad4f3d0f500\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:42:52.231725 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.231699    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d724fe-3d4a-493e-9783-3ad4f3d0f500-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dtmjd\" (UID: \"a7d724fe-3d4a-493e-9783-3ad4f3d0f500\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:42:52.292275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.292238    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjgjj\" (UniqueName: \"kubernetes.io/projected/a7d724fe-3d4a-493e-9783-3ad4f3d0f500-kube-api-access-pjgjj\") pod \"payload-pre-processing-7b98bb678c-dtmjd\" (UID: \"a7d724fe-3d4a-493e-9783-3ad4f3d0f500\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:42:52.313895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.313859    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:42:52.331349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.331319    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bd54dd-f5ec-4a45-b1fc-8534899e9b64-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fv5bz\" (UID: \"a2bd54dd-f5ec-4a45-b1fc-8534899e9b64\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:42:52.331486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.331395    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mrsr\" (UniqueName: \"kubernetes.io/projected/a2bd54dd-f5ec-4a45-b1fc-8534899e9b64-kube-api-access-6mrsr\") pod \"payload-processing-55fbbbf9dc-fv5bz\" (UID: \"a2bd54dd-f5ec-4a45-b1fc-8534899e9b64\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:42:52.332347 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.332320    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bd54dd-f5ec-4a45-b1fc-8534899e9b64-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fv5bz\" (UID: \"a2bd54dd-f5ec-4a45-b1fc-8534899e9b64\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:42:52.395450 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.395348    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mrsr\" (UniqueName: \"kubernetes.io/projected/a2bd54dd-f5ec-4a45-b1fc-8534899e9b64-kube-api-access-6mrsr\") pod \"payload-processing-55fbbbf9dc-fv5bz\" (UID: \"a2bd54dd-f5ec-4a45-b1fc-8534899e9b64\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:42:52.431917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.431880    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:42:52.694910 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.694881    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"]
Nov 02 01:42:52.696133 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:42:52.696107    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d724fe_3d4a_493e_9783_3ad4f3d0f500.slice/crio-0928372e08d6dc2c0d6029d5efd73f7e34499f279efdee605e0b5bfee9bd6ac4 WatchSource:0}: Error finding container 0928372e08d6dc2c0d6029d5efd73f7e34499f279efdee605e0b5bfee9bd6ac4: Status 404 returned error can't find the container with id 0928372e08d6dc2c0d6029d5efd73f7e34499f279efdee605e0b5bfee9bd6ac4
Nov 02 01:42:52.810449 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:52.810422    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"]
Nov 02 01:42:52.813065 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:42:52.813018    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bd54dd_f5ec_4a45_b1fc_8534899e9b64.slice/crio-d92fea6eb4461c8750c31ed5c48ad3116a1f9ed00c9eb62afbfa45a5ed3a641d WatchSource:0}: Error finding container d92fea6eb4461c8750c31ed5c48ad3116a1f9ed00c9eb62afbfa45a5ed3a641d: Status 404 returned error can't find the container with id d92fea6eb4461c8750c31ed5c48ad3116a1f9ed00c9eb62afbfa45a5ed3a641d
Nov 02 01:42:53.270665 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:53.270622    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd" event={"ID":"a7d724fe-3d4a-493e-9783-3ad4f3d0f500","Type":"ContainerStarted","Data":"0928372e08d6dc2c0d6029d5efd73f7e34499f279efdee605e0b5bfee9bd6ac4"}
Nov 02 01:42:53.272165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:53.272140    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz" event={"ID":"a2bd54dd-f5ec-4a45-b1fc-8534899e9b64","Type":"ContainerStarted","Data":"d92fea6eb4461c8750c31ed5c48ad3116a1f9ed00c9eb62afbfa45a5ed3a641d"}
Nov 02 01:42:56.297298 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:56.297255    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz" event={"ID":"a2bd54dd-f5ec-4a45-b1fc-8534899e9b64","Type":"ContainerStarted","Data":"ec0909d35107e7ba2cecea76c26a7d013a76d7f7401af161807224980017ce01"}
Nov 02 01:42:56.297792 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:56.297328    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:42:56.298911 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:56.298887    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd" event={"ID":"a7d724fe-3d4a-493e-9783-3ad4f3d0f500","Type":"ContainerStarted","Data":"2d015b1fc1e1dcf5080f403af6adb91200664d7fab7f6e1943c915fade69d685"}
Nov 02 01:42:56.298998 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:56.298950    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:42:56.413735 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:56.413677    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz" podStartSLOduration=1.421541247 podStartE2EDuration="4.413660716s" totalImagesPullingTime="2.992119469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:56.412881136 +0000 UTC m=+669.318346495" watchObservedRunningTime="2026-11-02 01:42:56.413660716 +0000 UTC m=+669.319126067"
Nov 02 01:42:56.530319 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:42:56.530260    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd" podStartSLOduration=2.332110296 podStartE2EDuration="5.530245438s" totalImagesPullingTime="3.198135142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:56.528110532 +0000 UTC m=+669.433575891" watchObservedRunningTime="2026-11-02 01:42:56.530245438 +0000 UTC m=+669.435710795"
Nov 02 01:43:07.307355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:07.307274    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dtmjd"
Nov 02 01:43:07.307355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:07.307349    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fv5bz"
Nov 02 01:43:12.136823 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:43:12.136787    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:43:14.458636 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:14.458598    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:14.991595 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:14.991563    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:16.610174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:16.610134    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:16.806248 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:16.806205    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:18.402836 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:18.402801    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:18.681734 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:18.681652    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:22.983455 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:22.983416    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:41.495972 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:41.495936    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:42.136675 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:43:42.136643    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:43:47.995519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:47.995474    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:48.328965 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.328873    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:48.619721 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.619694    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"]
Nov 02 01:43:48.623567 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.623540    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.692421 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.692382    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.692613 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.692437    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.692613 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.692542    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l48sq\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-kube-api-access-l48sq\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.692613 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.692582    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.692741 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.692634    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50954522-828f-43be-9e87-b281ad092a10-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.692794 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.692768    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.692836 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.692821    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50954522-828f-43be-9e87-b281ad092a10-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.692881 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.692868    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.692915 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.692896    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.706162 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.706140    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9470a3-openshift-default-dockercfg-pfjf7\""
Nov 02 01:43:48.749404 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.749365    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"]
Nov 02 01:43:48.793538 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.793501    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.793799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.793575    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.793799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.793613    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.793799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.793641    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.793799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.793666    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l48sq\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-kube-api-access-l48sq\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.793799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.793687    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.793799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.793738    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50954522-828f-43be-9e87-b281ad092a10-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.794178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.793804    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.794178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.793831    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50954522-828f-43be-9e87-b281ad092a10-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.794178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.794110    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.794178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.794133    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.794359 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.794222    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.794359 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.794303    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.794628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.794602    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50954522-828f-43be-9e87-b281ad092a10-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.796645 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.796622    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.796742 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.796724    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50954522-828f-43be-9e87-b281ad092a10-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.846852 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.846806    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:48.856573 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.856518    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l48sq\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-kube-api-access-l48sq\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.883849 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.883779    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:48.937491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:48.937455    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:49.113136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.113093    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:49.321107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.320996    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"]
Nov 02 01:43:49.328937 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.328915    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.334151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.334119    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:49.378416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.378391    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"]
Nov 02 01:43:49.379477 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:43:49.379450    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50954522_828f_43be_9e87_b281ad092a10.slice/crio-dbcba19113b704a9219e9c0705f5b4469a11efa7dd2c6176943ec6525744e35b WatchSource:0}: Error finding container dbcba19113b704a9219e9c0705f5b4469a11efa7dd2c6176943ec6525744e35b: Status 404 returned error can't find the container with id dbcba19113b704a9219e9c0705f5b4469a11efa7dd2c6176943ec6525744e35b
Nov 02 01:43:49.381724 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.381695    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:43:49.381796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.381761    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:43:49.381796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.381791    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:43:49.389788 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.389763    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"]
Nov 02 01:43:49.399006 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.398984    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.399124 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.399021    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvwq\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-kube-api-access-rvvwq\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.399124 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.399048    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.399208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.399140    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.399208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.399173    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.399297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.399242    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.399297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.399264    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.399396 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.399340    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.399475 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.399454    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.447626 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.447605    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ed0600-openshift-default-dockercfg-dhrkb\""
Nov 02 01:43:49.503447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.503097    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.503447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.503166    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.503447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.503203    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvwq\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-kube-api-access-rvvwq\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.503447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.503237    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.503447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.503277    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.503447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.503302    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.503447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.503358    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.503447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.503386    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.503447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.503417    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.504141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.504110    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.504141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.504125    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.505936 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.504362    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.505936 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.504584    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.505936 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.505363    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.507540 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.507519    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.507846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.507827    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.555012 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.554977    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:49.565931 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.565905    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.566282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.566265    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvwq\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-kube-api-access-rvvwq\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.597572 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.597496    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" event={"ID":"50954522-828f-43be-9e87-b281ad092a10","Type":"ContainerStarted","Data":"f46025c8d4701f51f1946451349886f4ac831953cf4c6a02dc4ad444c1ce0b75"}
Nov 02 01:43:49.597572 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.597550    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" event={"ID":"50954522-828f-43be-9e87-b281ad092a10","Type":"ContainerStarted","Data":"dbcba19113b704a9219e9c0705f5b4469a11efa7dd2c6176943ec6525744e35b"}
Nov 02 01:43:49.643403 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.643379    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:49.733465 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.733399    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" podStartSLOduration=1.733379199 podStartE2EDuration="1.733379199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:49.730738875 +0000 UTC m=+722.636204230" watchObservedRunningTime="2026-11-02 01:43:49.733379199 +0000 UTC m=+722.638844557"
Nov 02 01:43:49.821621 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.821592    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:49.938357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.938325    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:49.994832 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.994801    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"]
Nov 02 01:43:49.999711 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:49.999690    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.010802 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.010775    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.010927 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.010820    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4mq\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-kube-api-access-dl4mq\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.010927 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.010904    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.011029 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.010933    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44774b42-edee-4b78-b80c-70272d3af170-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.011117 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.011089    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.011181 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.011139    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.011240 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.011198    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.011294 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.011238    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44774b42-edee-4b78-b80c-70272d3af170-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.011338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.011292    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.026361 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.026334    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:50.038093 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.037476    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"]
Nov 02 01:43:50.041197 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:43:50.041170    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b205f3_b937_4bd5_80f8_3c2982a5fd3e.slice/crio-7516cab8d3155d684bb8f27896d6b3ffd88361c2f5c892c6b9fa1de0df6154a5 WatchSource:0}: Error finding container 7516cab8d3155d684bb8f27896d6b3ffd88361c2f5c892c6b9fa1de0df6154a5: Status 404 returned error can't find the container with id 7516cab8d3155d684bb8f27896d6b3ffd88361c2f5c892c6b9fa1de0df6154a5
Nov 02 01:43:50.043178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.043144    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:43:50.043310 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.043249    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:43:50.043310 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.043292    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:43:50.057174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.057117    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-55bc84-openshift-default-dockercfg-7g4tr\""
Nov 02 01:43:50.065858 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.065836    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"]
Nov 02 01:43:50.112106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.111969    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112002    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112019    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44774b42-edee-4b78-b80c-70272d3af170-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112094    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112215    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112244    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4mq\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-kube-api-access-dl4mq\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112276    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112302    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44774b42-edee-4b78-b80c-70272d3af170-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112551 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112354    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112551 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112402    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112551 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112485    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112570    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.112878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112657    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.113047 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.112986    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44774b42-edee-4b78-b80c-70272d3af170-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.114580 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.114559    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44774b42-edee-4b78-b80c-70272d3af170-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.114990 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.114857    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.173439 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.173405    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4mq\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-kube-api-access-dl4mq\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.173587 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.173407    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.213697 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.213611    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:50.315256 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.315220    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:50.398390 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.398286    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:50.576499 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.576404    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:50.605801 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.605756    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" event={"ID":"10b205f3-b937-4bd5-80f8-3c2982a5fd3e","Type":"ContainerStarted","Data":"c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa"}
Nov 02 01:43:50.606003 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.605829    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" event={"ID":"10b205f3-b937-4bd5-80f8-3c2982a5fd3e","Type":"ContainerStarted","Data":"7516cab8d3155d684bb8f27896d6b3ffd88361c2f5c892c6b9fa1de0df6154a5"}
Nov 02 01:43:50.644072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.644037    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:50.729980 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.729953    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"]
Nov 02 01:43:50.732373 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:43:50.732345    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44774b42_edee_4b78_b80c_70272d3af170.slice/crio-b4f127efdb55a1be1a09afed72ccbb45bc1e35223e2d598846a76e55e875b2d2 WatchSource:0}: Error finding container b4f127efdb55a1be1a09afed72ccbb45bc1e35223e2d598846a76e55e875b2d2: Status 404 returned error can't find the container with id b4f127efdb55a1be1a09afed72ccbb45bc1e35223e2d598846a76e55e875b2d2
Nov 02 01:43:50.735106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.735049    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:43:50.735208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.735146    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:43:50.735208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.735177    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:43:50.735670 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.735630    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" podStartSLOduration=1.735614758 podStartE2EDuration="1.735614758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:50.732980356 +0000 UTC m=+723.638445715" watchObservedRunningTime="2026-11-02 01:43:50.735614758 +0000 UTC m=+723.641080115"
Nov 02 01:43:50.764357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.764328    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:50.943478 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.943441    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:50.953715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:50.953680    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:51.613127 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:51.613093    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" event={"ID":"44774b42-edee-4b78-b80c-70272d3af170","Type":"ContainerStarted","Data":"3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c"}
Nov 02 01:43:51.613602 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:51.613143    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" event={"ID":"44774b42-edee-4b78-b80c-70272d3af170","Type":"ContainerStarted","Data":"b4f127efdb55a1be1a09afed72ccbb45bc1e35223e2d598846a76e55e875b2d2"}
Nov 02 01:43:51.613683 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:51.613634    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:51.614432 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:51.614414    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:43:51.648730 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:51.648702    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:51.736136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:51.736081    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" podStartSLOduration=2.736046067 podStartE2EDuration="2.736046067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:51.735252036 +0000 UTC m=+724.640717384" watchObservedRunningTime="2026-11-02 01:43:51.736046067 +0000 UTC m=+724.641511424"
Nov 02 01:43:52.316169 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:52.316139    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:52.321383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:52.321359    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:52.619975 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:52.619946    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:52.620403 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:52.619992    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:52.621186 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:52.621165    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:43:52.621303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:52.621220    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:43:53.077247 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:53.077165    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:53.297324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:53.297291    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:54.213284 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:54.213244    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:54.398497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:54.398458    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:54.578026 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:54.577930    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:54.854554 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:54.854472    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:55.046483 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:55.046445    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:55.231395 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:55.231368    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:55.499886 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:55.499800    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:55.680710 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:55.680675    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:57.237646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:57.237605    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:57.430839 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:57.430799    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:57.741718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:57.741675    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:57.927162 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:57.927130    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:58.111085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.110987    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:58.134404 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.134373    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"]
Nov 02 01:43:58.139414 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.139394    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:43:58.195822 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.195794    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"]
Nov 02 01:43:58.201883 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.201854    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7\" (UID: \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:43:58.202036 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.201986    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbzkm\" (UniqueName: \"kubernetes.io/projected/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-kube-api-access-pbzkm\") pod \"payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7\" (UID: \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:43:58.255091 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.253459    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9470a3\""
Nov 02 01:43:58.255091 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.253572    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9470a3-dockercfg-tdjdh\""
Nov 02 01:43:58.302701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.302670    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7\" (UID: \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:43:58.302913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.302889    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbzkm\" (UniqueName: \"kubernetes.io/projected/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-kube-api-access-pbzkm\") pod \"payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7\" (UID: \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:43:58.303453 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.303427    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7\" (UID: \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:43:58.365819 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.365793    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbzkm\" (UniqueName: \"kubernetes.io/projected/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-kube-api-access-pbzkm\") pod \"payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7\" (UID: \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:43:58.453519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.453490    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:43:58.491676 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.491642    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:58.682916 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.682837    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:58.829489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.829456    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"]
Nov 02 01:43:58.830980 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:43:58.830953    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039a37fd_6287_4fb0_b587_dc6fe5bcd7f2.slice/crio-56cbd9e5ebd98420f6d8fa1f966acb48bbd432f6148478e10a753e32127b2af7 WatchSource:0}: Error finding container 56cbd9e5ebd98420f6d8fa1f966acb48bbd432f6148478e10a753e32127b2af7: Status 404 returned error can't find the container with id 56cbd9e5ebd98420f6d8fa1f966acb48bbd432f6148478e10a753e32127b2af7
Nov 02 01:43:58.862634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:58.862605    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:43:59.665008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:59.664968    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7" event={"ID":"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2","Type":"ContainerStarted","Data":"7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223"}
Nov 02 01:43:59.665440 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:59.665024    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7" event={"ID":"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2","Type":"ContainerStarted","Data":"56cbd9e5ebd98420f6d8fa1f966acb48bbd432f6148478e10a753e32127b2af7"}
Nov 02 01:43:59.665440 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:59.665092    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:43:59.784905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:43:59.784853    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7" podStartSLOduration=1.784839163 podStartE2EDuration="1.784839163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:59.782871338 +0000 UTC m=+732.688336713" watchObservedRunningTime="2026-11-02 01:43:59.784839163 +0000 UTC m=+732.690304520"
Nov 02 01:44:02.670761 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:02.670731    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:02.859350 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:02.859312    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:03.078529 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.078450    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"]
Nov 02 01:44:03.082278 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.082262    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:03.153664 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.153627    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa22154-c3f8-47d0-8523-109b05e3a993-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk\" (UID: \"2fa22154-c3f8-47d0-8523-109b05e3a993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:03.153875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.153832    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgdc\" (UniqueName: \"kubernetes.io/projected/2fa22154-c3f8-47d0-8523-109b05e3a993-kube-api-access-7tgdc\") pod \"payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk\" (UID: \"2fa22154-c3f8-47d0-8523-109b05e3a993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:03.188271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.188237    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ed0600\""
Nov 02 01:44:03.188566 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.188542    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ed0600-dockercfg-fdv5k\""
Nov 02 01:44:03.194492 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.194461    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"]
Nov 02 01:44:03.254866 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.254825    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa22154-c3f8-47d0-8523-109b05e3a993-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk\" (UID: \"2fa22154-c3f8-47d0-8523-109b05e3a993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:03.255026 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.254909    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgdc\" (UniqueName: \"kubernetes.io/projected/2fa22154-c3f8-47d0-8523-109b05e3a993-kube-api-access-7tgdc\") pod \"payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk\" (UID: \"2fa22154-c3f8-47d0-8523-109b05e3a993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:03.255754 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.255734    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa22154-c3f8-47d0-8523-109b05e3a993-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk\" (UID: \"2fa22154-c3f8-47d0-8523-109b05e3a993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:03.317647 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.317612    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgdc\" (UniqueName: \"kubernetes.io/projected/2fa22154-c3f8-47d0-8523-109b05e3a993-kube-api-access-7tgdc\") pod \"payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk\" (UID: \"2fa22154-c3f8-47d0-8523-109b05e3a993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:03.396347 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.396316    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:03.772863 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:03.772838    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"]
Nov 02 01:44:03.774546 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:44:03.774518    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa22154_c3f8_47d0_8523_109b05e3a993.slice/crio-7da395eca1435fab67f42cc08caabfe38ed10631cfd10baba58da4f31a29e492 WatchSource:0}: Error finding container 7da395eca1435fab67f42cc08caabfe38ed10631cfd10baba58da4f31a29e492: Status 404 returned error can't find the container with id 7da395eca1435fab67f42cc08caabfe38ed10631cfd10baba58da4f31a29e492
Nov 02 01:44:04.589251 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:04.589212    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:04.697777 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:04.697744    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk" event={"ID":"2fa22154-c3f8-47d0-8523-109b05e3a993","Type":"ContainerStarted","Data":"997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c"}
Nov 02 01:44:04.697970 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:04.697793    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk" event={"ID":"2fa22154-c3f8-47d0-8523-109b05e3a993","Type":"ContainerStarted","Data":"7da395eca1435fab67f42cc08caabfe38ed10631cfd10baba58da4f31a29e492"}
Nov 02 01:44:04.697970 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:04.697823    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:04.815304 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:04.815251    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk" podStartSLOduration=1.815237722 podStartE2EDuration="1.815237722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:04.81261923 +0000 UTC m=+737.718084587" watchObservedRunningTime="2026-11-02 01:44:04.815237722 +0000 UTC m=+737.720703078"
Nov 02 01:44:04.822803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:04.822777    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:05.228999 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.228960    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:05.418146 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.418107    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:05.609198 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.609115    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"]
Nov 02 01:44:05.612941 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.612926    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:05.678731 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.678693    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljh5b\" (UniqueName: \"kubernetes.io/projected/e9e74e12-b3db-4d35-9228-2a03fd7315d4-kube-api-access-ljh5b\") pod \"payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c\" (UID: \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:05.678963 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.678934    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e74e12-b3db-4d35-9228-2a03fd7315d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c\" (UID: \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:05.722726 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.722623    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-0ebdb3\""
Nov 02 01:44:05.722726 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.722719    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-0ebdb3-dockercfg-7ckk2\""
Nov 02 01:44:05.725334 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.725306    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"]
Nov 02 01:44:05.780529 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.780483    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljh5b\" (UniqueName: \"kubernetes.io/projected/e9e74e12-b3db-4d35-9228-2a03fd7315d4-kube-api-access-ljh5b\") pod \"payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c\" (UID: \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:05.780748 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.780723    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e74e12-b3db-4d35-9228-2a03fd7315d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c\" (UID: \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:05.781414 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.781393    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e74e12-b3db-4d35-9228-2a03fd7315d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c\" (UID: \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:05.841370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.841339    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljh5b\" (UniqueName: \"kubernetes.io/projected/e9e74e12-b3db-4d35-9228-2a03fd7315d4-kube-api-access-ljh5b\") pod \"payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c\" (UID: \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:05.926902 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:05.926872    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:06.302410 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:06.302381    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"]
Nov 02 01:44:06.303662 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:44:06.303637    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e74e12_b3db_4d35_9228_2a03fd7315d4.slice/crio-f55082dba3103e9f7c4898b043f3b73f7bcae8ef19bc4566622e83fcb1eec612 WatchSource:0}: Error finding container f55082dba3103e9f7c4898b043f3b73f7bcae8ef19bc4566622e83fcb1eec612: Status 404 returned error can't find the container with id f55082dba3103e9f7c4898b043f3b73f7bcae8ef19bc4566622e83fcb1eec612
Nov 02 01:44:06.734946 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:06.734911    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c" event={"ID":"e9e74e12-b3db-4d35-9228-2a03fd7315d4","Type":"ContainerStarted","Data":"c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146"}
Nov 02 01:44:06.735146 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:06.734962    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c" event={"ID":"e9e74e12-b3db-4d35-9228-2a03fd7315d4","Type":"ContainerStarted","Data":"f55082dba3103e9f7c4898b043f3b73f7bcae8ef19bc4566622e83fcb1eec612"}
Nov 02 01:44:06.735146 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:06.735045    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:06.850610 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:06.850550    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c" podStartSLOduration=1.8505323040000001 podStartE2EDuration="1.850532304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:06.850138067 +0000 UTC m=+739.755603423" watchObservedRunningTime="2026-11-02 01:44:06.850532304 +0000 UTC m=+739.755997661"
Nov 02 01:44:09.909716 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:09.909681    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:10.104381 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:10.104318    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:10.286360 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:10.286268    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:10.674938 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:10.674911    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 01:44:12.137686 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:44:12.137649    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:44:12.419000 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:12.418924    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:12.651174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:12.651138    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:12.807835 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:12.807759    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"]
Nov 02 01:44:12.812691 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:12.812673    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:12.916465 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:12.916435    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-061fab\""
Nov 02 01:44:12.916465 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:12.916447    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-061fab-dockercfg-hqmkq\""
Nov 02 01:44:12.919309 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:12.919291    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"]
Nov 02 01:44:12.976109 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:12.970508    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc0224f-76d8-4279-8739-f6165803bd86-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n\" (UID: \"efc0224f-76d8-4279-8739-f6165803bd86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:12.976109 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:12.970601    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6fj7\" (UniqueName: \"kubernetes.io/projected/efc0224f-76d8-4279-8739-f6165803bd86-kube-api-access-z6fj7\") pod \"payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n\" (UID: \"efc0224f-76d8-4279-8739-f6165803bd86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:13.073951 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:13.073875    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc0224f-76d8-4279-8739-f6165803bd86-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n\" (UID: \"efc0224f-76d8-4279-8739-f6165803bd86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:13.073951 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:13.073923    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6fj7\" (UniqueName: \"kubernetes.io/projected/efc0224f-76d8-4279-8739-f6165803bd86-kube-api-access-z6fj7\") pod \"payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n\" (UID: \"efc0224f-76d8-4279-8739-f6165803bd86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:13.074579 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:13.074557    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc0224f-76d8-4279-8739-f6165803bd86-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n\" (UID: \"efc0224f-76d8-4279-8739-f6165803bd86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:13.135673 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:13.135645    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6fj7\" (UniqueName: \"kubernetes.io/projected/efc0224f-76d8-4279-8739-f6165803bd86-kube-api-access-z6fj7\") pod \"payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n\" (UID: \"efc0224f-76d8-4279-8739-f6165803bd86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:13.425996 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:13.425964    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:13.803036 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:13.803008    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"]
Nov 02 01:44:13.805927 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:44:13.805879    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc0224f_76d8_4279_8739_f6165803bd86.slice/crio-dfd1501d5fe574ce2c4aa338f42c674686e5578acfa59935f3c731484d0397cb WatchSource:0}: Error finding container dfd1501d5fe574ce2c4aa338f42c674686e5578acfa59935f3c731484d0397cb: Status 404 returned error can't find the container with id dfd1501d5fe574ce2c4aa338f42c674686e5578acfa59935f3c731484d0397cb
Nov 02 01:44:14.791272 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:14.791222    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:14.796521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:14.796490    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n" event={"ID":"efc0224f-76d8-4279-8739-f6165803bd86","Type":"ContainerStarted","Data":"6fbec0648f9cab7ff3043fdf527884e44daf1e88fa456b6dd6c436c3120d499d"}
Nov 02 01:44:14.796685 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:14.796543    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n" event={"ID":"efc0224f-76d8-4279-8739-f6165803bd86","Type":"ContainerStarted","Data":"dfd1501d5fe574ce2c4aa338f42c674686e5578acfa59935f3c731484d0397cb"}
Nov 02 01:44:14.796685 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:14.796587    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:14.913902 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:14.913850    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n" podStartSLOduration=2.913835682 podStartE2EDuration="2.913835682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:14.91145274 +0000 UTC m=+747.816918093" watchObservedRunningTime="2026-11-02 01:44:14.913835682 +0000 UTC m=+747.819301037"
Nov 02 01:44:15.028153 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.028121    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:15.205076 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.205036    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:15.320317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.320289    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"]
Nov 02 01:44:15.323877 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.323856    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:15.388610 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.388569    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"]
Nov 02 01:44:15.428166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.428136    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-55bc84-dockercfg-lxlx7\""
Nov 02 01:44:15.428166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.428136    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-55bc84\""
Nov 02 01:44:15.477286 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.477202    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"]
Nov 02 01:44:15.481101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.481036    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:15.509158 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.509126    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74frn\" (UniqueName: \"kubernetes.io/projected/633f0926-b18e-4ed5-b916-268160edee9a-kube-api-access-74frn\") pod \"payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx\" (UID: \"633f0926-b18e-4ed5-b916-268160edee9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:15.509284 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.509177    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633f0926-b18e-4ed5-b916-268160edee9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx\" (UID: \"633f0926-b18e-4ed5-b916-268160edee9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:15.524849 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.524824    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"]
Nov 02 01:44:15.610685 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.610642    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xjz\" (UniqueName: \"kubernetes.io/projected/956e87a3-da69-4672-82ec-772ed56c413a-kube-api-access-d8xjz\") pod \"payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj\" (UID: \"956e87a3-da69-4672-82ec-772ed56c413a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:15.610898 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.610835    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74frn\" (UniqueName: \"kubernetes.io/projected/633f0926-b18e-4ed5-b916-268160edee9a-kube-api-access-74frn\") pod \"payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx\" (UID: \"633f0926-b18e-4ed5-b916-268160edee9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:15.610898 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.610890    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633f0926-b18e-4ed5-b916-268160edee9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx\" (UID: \"633f0926-b18e-4ed5-b916-268160edee9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:15.611012 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.610916    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956e87a3-da69-4672-82ec-772ed56c413a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj\" (UID: \"956e87a3-da69-4672-82ec-772ed56c413a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:15.611644 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.611618    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633f0926-b18e-4ed5-b916-268160edee9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx\" (UID: \"633f0926-b18e-4ed5-b916-268160edee9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:15.676423 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.676397    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74frn\" (UniqueName: \"kubernetes.io/projected/633f0926-b18e-4ed5-b916-268160edee9a-kube-api-access-74frn\") pod \"payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx\" (UID: \"633f0926-b18e-4ed5-b916-268160edee9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:15.712294 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.712261    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956e87a3-da69-4672-82ec-772ed56c413a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj\" (UID: \"956e87a3-da69-4672-82ec-772ed56c413a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:15.712481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.712328    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xjz\" (UniqueName: \"kubernetes.io/projected/956e87a3-da69-4672-82ec-772ed56c413a-kube-api-access-d8xjz\") pod \"payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj\" (UID: \"956e87a3-da69-4672-82ec-772ed56c413a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:15.712962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.712937    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956e87a3-da69-4672-82ec-772ed56c413a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj\" (UID: \"956e87a3-da69-4672-82ec-772ed56c413a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:15.730583 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.730522    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 01:44:15.777326 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.777288    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xjz\" (UniqueName: \"kubernetes.io/projected/956e87a3-da69-4672-82ec-772ed56c413a-kube-api-access-d8xjz\") pod \"payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj\" (UID: \"956e87a3-da69-4672-82ec-772ed56c413a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:15.793539 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.793515    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:15.937518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:15.937490    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:16.200708 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:16.200672    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"]
Nov 02 01:44:16.374861 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:16.371550    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"]
Nov 02 01:44:16.376022 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:44:16.375995    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633f0926_b18e_4ed5_b916_268160edee9a.slice/crio-68e5a38df6282e4158e03d747119c53a902fc94426bc166bd40dbc62a0edc543 WatchSource:0}: Error finding container 68e5a38df6282e4158e03d747119c53a902fc94426bc166bd40dbc62a0edc543: Status 404 returned error can't find the container with id 68e5a38df6282e4158e03d747119c53a902fc94426bc166bd40dbc62a0edc543
Nov 02 01:44:16.810740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:16.810697    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj" event={"ID":"956e87a3-da69-4672-82ec-772ed56c413a","Type":"ContainerStarted","Data":"03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290"}
Nov 02 01:44:16.811267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:16.810766    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:16.811267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:16.810791    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj" event={"ID":"956e87a3-da69-4672-82ec-772ed56c413a","Type":"ContainerStarted","Data":"d3c81cfc112cc27faa42e8a93f367c3639283b34a3a237ce7ac9017f6ea2fc6b"}
Nov 02 01:44:16.812322 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:16.812301    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx" event={"ID":"633f0926-b18e-4ed5-b916-268160edee9a","Type":"ContainerStarted","Data":"bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85"}
Nov 02 01:44:16.812420 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:16.812337    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx" event={"ID":"633f0926-b18e-4ed5-b916-268160edee9a","Type":"ContainerStarted","Data":"68e5a38df6282e4158e03d747119c53a902fc94426bc166bd40dbc62a0edc543"}
Nov 02 01:44:16.812460 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:16.812424    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:16.934369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:16.934317    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj" podStartSLOduration=1.9343015430000001 podStartE2EDuration="1.934301543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:16.932790408 +0000 UTC m=+749.838255767" watchObservedRunningTime="2026-11-02 01:44:16.934301543 +0000 UTC m=+749.839766900"
Nov 02 01:44:17.071145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:17.069829    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx" podStartSLOduration=2.069806714 podStartE2EDuration="2.069806714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:17.05643056 +0000 UTC m=+749.961895921" watchObservedRunningTime="2026-11-02 01:44:17.069806714 +0000 UTC m=+749.975272070"
Nov 02 01:44:17.744528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:17.744499    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 01:44:19.454763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.454728    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"]
Nov 02 01:44:19.458655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.458637    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.511102 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.511044    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ed0600-serving-cert\""
Nov 02 01:44:19.551954 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.551924    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c785f0a-71f7-432c-900f-bb08d6b82636-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.552163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.551965    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.552163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.552137    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-metrics-certs\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.552293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.552249    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djv8\" (UniqueName: \"kubernetes.io/projected/5c785f0a-71f7-432c-900f-bb08d6b82636-kube-api-access-6djv8\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.560374 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.560357    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 01:44:19.560467 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.560361    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2n4c5\""
Nov 02 01:44:19.563370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.563347    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"]
Nov 02 01:44:19.653123 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.653052    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djv8\" (UniqueName: \"kubernetes.io/projected/5c785f0a-71f7-432c-900f-bb08d6b82636-kube-api-access-6djv8\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.653297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.653160    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c785f0a-71f7-432c-900f-bb08d6b82636-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.653297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.653204    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.653416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.653317    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-metrics-certs\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.653416 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:44:19.653399    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-serving-cert: secret "maas-api-e2e-worker-w4-ed0600-serving-cert" not found
Nov 02 01:44:19.653524 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:44:19.653467    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls podName:5c785f0a-71f7-432c-900f-bb08d6b82636 nodeName:}" failed. No retries permitted until 2026-11-02 01:44:20.153446336 +0000 UTC m=+753.058911671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls") pod "maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv" (UID: "5c785f0a-71f7-432c-900f-bb08d6b82636") : secret "maas-api-e2e-worker-w4-ed0600-serving-cert" not found
Nov 02 01:44:19.653824 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.653803    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c785f0a-71f7-432c-900f-bb08d6b82636-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.655805 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.655779    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-metrics-certs\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.677568 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.677531    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:19.713856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.713789    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djv8\" (UniqueName: \"kubernetes.io/projected/5c785f0a-71f7-432c-900f-bb08d6b82636-kube-api-access-6djv8\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:19.905495 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:19.905427    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:20.084982 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:20.084887    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:20.158505 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:20.158474    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:20.161152 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:20.161130    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:20.372600 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:20.372560    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:20.760772 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:20.760745    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"]
Nov 02 01:44:20.761640 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:44:20.761606    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c785f0a_71f7_432c_900f_bb08d6b82636.slice/crio-cef2abf5ccdaff2a1b9f714aa2e4e91e35b5b48253cd44d9265ee65cd8849664 WatchSource:0}: Error finding container cef2abf5ccdaff2a1b9f714aa2e4e91e35b5b48253cd44d9265ee65cd8849664: Status 404 returned error can't find the container with id cef2abf5ccdaff2a1b9f714aa2e4e91e35b5b48253cd44d9265ee65cd8849664
Nov 02 01:44:20.837531 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:20.837491    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv" event={"ID":"5c785f0a-71f7-432c-900f-bb08d6b82636","Type":"ContainerStarted","Data":"cef2abf5ccdaff2a1b9f714aa2e4e91e35b5b48253cd44d9265ee65cd8849664"}
Nov 02 01:44:22.278646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.278603    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:22.498921 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.498885    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:22.636158 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.636045    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:44:22.636305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.636196    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:44:22.816591 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.816514    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"]
Nov 02 01:44:22.821810 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.821353    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:22.855655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.855628    4912 generic.go:356] "Generic (PLEG): container finished" podID="50954522-828f-43be-9e87-b281ad092a10" containerID="f46025c8d4701f51f1946451349886f4ac831953cf4c6a02dc4ad444c1ce0b75" exitCode=0
Nov 02 01:44:22.855813 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.855668    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" event={"ID":"50954522-828f-43be-9e87-b281ad092a10","Type":"ContainerDied","Data":"f46025c8d4701f51f1946451349886f4ac831953cf4c6a02dc4ad444c1ce0b75"}
Nov 02 01:44:22.856035 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.856021    4912 scope.go:122] "RemoveContainer" containerID="f46025c8d4701f51f1946451349886f4ac831953cf4c6a02dc4ad444c1ce0b75"
Nov 02 01:44:22.857595 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.857561    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:44:22.857701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.857651    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:44:22.857764 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.857698    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:44:22.882299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.878606    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"]
Nov 02 01:44:22.905332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.904870    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b93040-c359-4e41-a8ff-923846973e4a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69\" (UID: \"59b93040-c359-4e41-a8ff-923846973e4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:22.905332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.904985    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgc4\" (UniqueName: \"kubernetes.io/projected/59b93040-c359-4e41-a8ff-923846973e4a-kube-api-access-9tgc4\") pod \"payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69\" (UID: \"59b93040-c359-4e41-a8ff-923846973e4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:22.922865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.922825    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-dbe849-dockercfg-qmm9r\""
Nov 02 01:44:22.922865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:22.922834    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-dbe849\""
Nov 02 01:44:23.007519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.006693    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgc4\" (UniqueName: \"kubernetes.io/projected/59b93040-c359-4e41-a8ff-923846973e4a-kube-api-access-9tgc4\") pod \"payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69\" (UID: \"59b93040-c359-4e41-a8ff-923846973e4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:23.007519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.006953    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b93040-c359-4e41-a8ff-923846973e4a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69\" (UID: \"59b93040-c359-4e41-a8ff-923846973e4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:23.008344 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.008289    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b93040-c359-4e41-a8ff-923846973e4a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69\" (UID: \"59b93040-c359-4e41-a8ff-923846973e4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:23.073300 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.073271    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgc4\" (UniqueName: \"kubernetes.io/projected/59b93040-c359-4e41-a8ff-923846973e4a-kube-api-access-9tgc4\") pod \"payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69\" (UID: \"59b93040-c359-4e41-a8ff-923846973e4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:23.136563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.136532    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:23.546800 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:44:23.546768    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b93040_c359_4e41_a8ff_923846973e4a.slice/crio-d27d0adf050a2c992e3d22bbd1e9dd4f5e3ceed2f932f3ab2be3595472934e38 WatchSource:0}: Error finding container d27d0adf050a2c992e3d22bbd1e9dd4f5e3ceed2f932f3ab2be3595472934e38: Status 404 returned error can't find the container with id d27d0adf050a2c992e3d22bbd1e9dd4f5e3ceed2f932f3ab2be3595472934e38
Nov 02 01:44:23.547509 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.547482    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"]
Nov 02 01:44:23.907916 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.906934    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" event={"ID":"50954522-828f-43be-9e87-b281ad092a10","Type":"ContainerStarted","Data":"56db471be5f5433a711062d6a5e648d4600cf0712a348acc088da87a21fb391e"}
Nov 02 01:44:23.911723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.911007    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69" event={"ID":"59b93040-c359-4e41-a8ff-923846973e4a","Type":"ContainerStarted","Data":"d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740"}
Nov 02 01:44:23.911723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.911086    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69" event={"ID":"59b93040-c359-4e41-a8ff-923846973e4a","Type":"ContainerStarted","Data":"d27d0adf050a2c992e3d22bbd1e9dd4f5e3ceed2f932f3ab2be3595472934e38"}
Nov 02 01:44:23.911723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.911671    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:23.938558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.938491    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:44:23.938558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.938508    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:44:23.938558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:23.938546    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 01:44:24.219341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:24.218949    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69" podStartSLOduration=2.218935809 podStartE2EDuration="2.218935809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:24.215345338 +0000 UTC m=+757.120810694" watchObservedRunningTime="2026-11-02 01:44:24.218935809 +0000 UTC m=+757.124401167"
Nov 02 01:44:24.680183 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:24.680092    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:24.938438 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:24.938364    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:44:24.938438 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:24.938417    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 01:44:25.806352 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:25.806271    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 01:44:25.931957 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:25.931910    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv" event={"ID":"5c785f0a-71f7-432c-900f-bb08d6b82636","Type":"ContainerStarted","Data":"5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb"}
Nov 02 01:44:25.932308 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:25.932277    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:25.938821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:25.938788    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:44:25.938974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:25.938843    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 01:44:26.092141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:26.078390    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv" podStartSLOduration=2.293296554 podStartE2EDuration="7.078372916s" totalImagesPullingTime="4.785076362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:26.076766732 +0000 UTC m=+758.982232088" watchObservedRunningTime="2026-11-02 01:44:26.078372916 +0000 UTC m=+758.983838274"
Nov 02 01:44:26.938008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:26.937975    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:44:26.938417 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:26.938027    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 01:44:27.820521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:27.820493    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 01:44:27.820725 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:27.820571    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 01:44:27.939096 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:27.938858    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:44:27.939096 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:27.938912    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 01:44:28.938491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:28.938458    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:44:28.938746 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:28.938723    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 01:44:29.418794 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:29.418755    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:29.654106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:29.654043    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:29.831913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:29.831804    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:29.942088 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:29.942039    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:44:29.942498 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:29.942480    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:44:29.943369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:29.943351    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 01:44:31.952466 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:31.952436    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 01:44:32.993653 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:32.993615    4912 generic.go:356] "Generic (PLEG): container finished" podID="44774b42-edee-4b78-b80c-70272d3af170" containerID="3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c" exitCode=0
Nov 02 01:44:32.994202 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:32.993674    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" event={"ID":"44774b42-edee-4b78-b80c-70272d3af170","Type":"ContainerDied","Data":"3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c"}
Nov 02 01:44:32.994202 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:32.994094    4912 scope.go:122] "RemoveContainer" containerID="3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c"
Nov 02 01:44:32.995559 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:32.995529    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:44:32.995658 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:32.995610    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:44:32.995658 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:32.995643    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:44:34.008072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:34.008029    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" event={"ID":"44774b42-edee-4b78-b80c-70272d3af170","Type":"ContainerStarted","Data":"ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c"}
Nov 02 01:44:34.315679 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:34.315566    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:44:34.316707 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:34.316684    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:44:34.316801 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:34.316737    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 02 01:44:34.328179 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:34.328154    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:34.509155 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:34.509113    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:34.693137 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:34.693095    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:35.316014 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:35.315963    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:44:35.316426 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:35.316035    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 02 01:44:35.935210 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:35.935179    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 01:44:36.316534 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:36.316451    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:44:36.316534 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:36.316518    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 02 01:44:37.315547 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:37.315513    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:44:37.315749 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:37.315571    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 02 01:44:37.619520 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:37.619487    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:44:38.316423 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:38.316394    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:44:38.316577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:38.316439    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 02 01:44:39.320643 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:39.320613    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:44:39.321826 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:39.321806    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 01:44:39.372768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:39.372726    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:41.850613 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:41.850576    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:42.137246 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:44:42.137208    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:44:43.068651 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:43.068616    4912 generic.go:356] "Generic (PLEG): container finished" podID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerID="c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa" exitCode=0
Nov 02 01:44:43.069182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:43.068689    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" event={"ID":"10b205f3-b937-4bd5-80f8-3c2982a5fd3e","Type":"ContainerDied","Data":"c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa"}
Nov 02 01:44:43.069182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:43.069105    4912 scope.go:122] "RemoveContainer" containerID="c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa"
Nov 02 01:44:43.070569 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:43.070537    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:44:43.070678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:43.070608    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:44:43.070678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:43.070649    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:44:44.087766 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:44.087735    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" event={"ID":"10b205f3-b937-4bd5-80f8-3c2982a5fd3e","Type":"ContainerStarted","Data":"24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058"}
Nov 02 01:44:44.176416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:44.176382    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:44.398261 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:44.398220    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:44.644634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:44.644600    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:44:44.644845 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:44.644802    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:44:44.644909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:44.644869    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 01:44:45.644387 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:45.644348    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:44:45.644797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:45.644402    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 01:44:46.644341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:46.644302    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:44:46.644548 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:46.644363    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 01:44:47.644219 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:47.644179    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:44:47.644419 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:47.644236    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 01:44:48.643738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:48.643697    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:44:48.644238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:48.643763    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 01:44:48.822247 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:48.822209    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:49.467527 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:49.467489    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:49.648823 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:49.648796    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:44:49.649271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:49.649242    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:44:49.650023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:49.650002    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 01:44:49.652259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:49.652235    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:53.463280 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:53.463233    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:55.135878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:55.135835    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:59.669443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:59.669403    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:44:59.845079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:44:59.845027    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:00.169872 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.169837    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"]
Nov 02 01:45:00.175163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.175138    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"]
Nov 02 01:45:00.175309 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.175257    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:00.178789 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.178769    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:00.199268 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.199247    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"]
Nov 02 01:45:00.202916 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.202902    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:00.219170 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.219150    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"]
Nov 02 01:45:00.222672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.222654    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:00.227022 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.227007    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:45:00.293673 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.293640    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"]
Nov 02 01:45:00.297228 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.297198    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"]
Nov 02 01:45:00.323382 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.323342    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"]
Nov 02 01:45:00.329143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.329110    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v\" (UID: \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:00.329314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.329164    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdp7\" (UniqueName: \"kubernetes.io/projected/1a8f3dae-1842-4851-abf2-593fd1c81f3a-kube-api-access-nvdp7\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd\" (UID: \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:00.329314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.329261    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8f4t\" (UniqueName: \"kubernetes.io/projected/3e67d151-dd2a-4191-8e5d-797bf2e9403f-kube-api-access-n8f4t\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd\" (UID: \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:00.329314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.329292    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e67d151-dd2a-4191-8e5d-797bf2e9403f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd\" (UID: \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:00.329501 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.329382    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzzgh\" (UniqueName: \"kubernetes.io/projected/00484af4-8fb8-42d0-918e-597f661cef06-kube-api-access-jzzgh\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq\" (UID: \"00484af4-8fb8-42d0-918e-597f661cef06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:00.329501 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.329428    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f3dae-1842-4851-abf2-593fd1c81f3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd\" (UID: \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:00.329611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.329519    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00484af4-8fb8-42d0-918e-597f661cef06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq\" (UID: \"00484af4-8fb8-42d0-918e-597f661cef06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:00.329611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.329560    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnv7\" (UniqueName: \"kubernetes.io/projected/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-kube-api-access-xfnv7\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v\" (UID: \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:00.334894 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.334863    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"]
Nov 02 01:45:00.379208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.379169    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:00.430329 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.430242    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnv7\" (UniqueName: \"kubernetes.io/projected/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-kube-api-access-xfnv7\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v\" (UID: \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:00.430329 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.430300    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v\" (UID: \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:00.430569 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.430464    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdp7\" (UniqueName: \"kubernetes.io/projected/1a8f3dae-1842-4851-abf2-593fd1c81f3a-kube-api-access-nvdp7\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd\" (UID: \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:00.430569 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.430549    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8f4t\" (UniqueName: \"kubernetes.io/projected/3e67d151-dd2a-4191-8e5d-797bf2e9403f-kube-api-access-n8f4t\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd\" (UID: \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:00.430696 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.430574    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e67d151-dd2a-4191-8e5d-797bf2e9403f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd\" (UID: \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:00.430696 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.430648    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzzgh\" (UniqueName: \"kubernetes.io/projected/00484af4-8fb8-42d0-918e-597f661cef06-kube-api-access-jzzgh\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq\" (UID: \"00484af4-8fb8-42d0-918e-597f661cef06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:00.430808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.430694    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f3dae-1842-4851-abf2-593fd1c81f3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd\" (UID: \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:00.430808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.430771    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00484af4-8fb8-42d0-918e-597f661cef06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq\" (UID: \"00484af4-8fb8-42d0-918e-597f661cef06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:00.431032 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.431011    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v\" (UID: \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:00.431258 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.431240    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e67d151-dd2a-4191-8e5d-797bf2e9403f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd\" (UID: \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:00.431427 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.431410    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00484af4-8fb8-42d0-918e-597f661cef06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq\" (UID: \"00484af4-8fb8-42d0-918e-597f661cef06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:00.431462 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.431434    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f3dae-1842-4851-abf2-593fd1c81f3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd\" (UID: \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:00.491571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.491528    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzzgh\" (UniqueName: \"kubernetes.io/projected/00484af4-8fb8-42d0-918e-597f661cef06-kube-api-access-jzzgh\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq\" (UID: \"00484af4-8fb8-42d0-918e-597f661cef06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:00.491718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.491570    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdp7\" (UniqueName: \"kubernetes.io/projected/1a8f3dae-1842-4851-abf2-593fd1c81f3a-kube-api-access-nvdp7\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd\" (UID: \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:00.491718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.491690    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8f4t\" (UniqueName: \"kubernetes.io/projected/3e67d151-dd2a-4191-8e5d-797bf2e9403f-kube-api-access-n8f4t\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd\" (UID: \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:00.491796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.491691    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnv7\" (UniqueName: \"kubernetes.io/projected/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-kube-api-access-xfnv7\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v\" (UID: \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:00.496330 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.496311    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:00.516273 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.516250    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:00.534795 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.534763    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:00.591723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.591646    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:00.790917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.790833    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:00.882837 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.882765    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"]
Nov 02 01:45:00.885948 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:45:00.885403    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f7b8bc_ead3_4eb9_a410_6bcc3aa6b071.slice/crio-6455a0f79d76e4c7b8ed6294cc3e3d3a6c5bb9acf103b70be65d4fb9e74a3493 WatchSource:0}: Error finding container 6455a0f79d76e4c7b8ed6294cc3e3d3a6c5bb9acf103b70be65d4fb9e74a3493: Status 404 returned error can't find the container with id 6455a0f79d76e4c7b8ed6294cc3e3d3a6c5bb9acf103b70be65d4fb9e74a3493
Nov 02 01:45:00.931957 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.931932    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"]
Nov 02 01:45:00.932608 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:45:00.932587    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8f3dae_1842_4851_abf2_593fd1c81f3a.slice/crio-a06e07856bdab8a93a3f65e0d097921da51910bd3a2241c63e8206462845b767 WatchSource:0}: Error finding container a06e07856bdab8a93a3f65e0d097921da51910bd3a2241c63e8206462845b767: Status 404 returned error can't find the container with id a06e07856bdab8a93a3f65e0d097921da51910bd3a2241c63e8206462845b767
Nov 02 01:45:00.968562 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:00.968534    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"]
Nov 02 01:45:00.968683 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:45:00.968663    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00484af4_8fb8_42d0_918e_597f661cef06.slice/crio-aac791660fbf1222cc8ab4139800c937af4e2010353cdbc73f0f105067a9f9a1 WatchSource:0}: Error finding container aac791660fbf1222cc8ab4139800c937af4e2010353cdbc73f0f105067a9f9a1: Status 404 returned error can't find the container with id aac791660fbf1222cc8ab4139800c937af4e2010353cdbc73f0f105067a9f9a1
Nov 02 01:45:01.181006 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:01.180921    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"]
Nov 02 01:45:01.183049 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:45:01.183025    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e67d151_dd2a_4191_8e5d_797bf2e9403f.slice/crio-bf4228fa8462968e866dc1b0e66db4110a83677c4f01d513a046ef133210ef99 WatchSource:0}: Error finding container bf4228fa8462968e866dc1b0e66db4110a83677c4f01d513a046ef133210ef99: Status 404 returned error can't find the container with id bf4228fa8462968e866dc1b0e66db4110a83677c4f01d513a046ef133210ef99
Nov 02 01:45:01.188521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:01.188495    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:01.205611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:01.205579    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v" event={"ID":"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071","Type":"ContainerStarted","Data":"6455a0f79d76e4c7b8ed6294cc3e3d3a6c5bb9acf103b70be65d4fb9e74a3493"}
Nov 02 01:45:01.207250 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:01.207225    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq" event={"ID":"00484af4-8fb8-42d0-918e-597f661cef06","Type":"ContainerStarted","Data":"aac791660fbf1222cc8ab4139800c937af4e2010353cdbc73f0f105067a9f9a1"}
Nov 02 01:45:01.208591 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:01.208569    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" event={"ID":"1a8f3dae-1842-4851-abf2-593fd1c81f3a","Type":"ContainerStarted","Data":"a06e07856bdab8a93a3f65e0d097921da51910bd3a2241c63e8206462845b767"}
Nov 02 01:45:01.209815 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:01.209796    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" event={"ID":"3e67d151-dd2a-4191-8e5d-797bf2e9403f","Type":"ContainerStarted","Data":"bf4228fa8462968e866dc1b0e66db4110a83677c4f01d513a046ef133210ef99"}
Nov 02 01:45:01.367322 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:01.367279    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:05.488563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:05.488524    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:05.987434 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:05.987395    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:06.263385 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:06.263345    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" event={"ID":"1a8f3dae-1842-4851-abf2-593fd1c81f3a","Type":"ContainerStarted","Data":"b052f45c28144ffd61238cbdc40b308d1e116230b44cff85df55a606a89079e9"}
Nov 02 01:45:06.379308 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:06.379260    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" podStartSLOduration=1.66715871 podStartE2EDuration="6.379244519s" totalImagesPullingTime="4.712085809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:06.376492954 +0000 UTC m=+799.281958338" watchObservedRunningTime="2026-11-02 01:45:06.379244519 +0000 UTC m=+799.284709875"
Nov 02 01:45:06.563270 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:06.563222    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:06.741384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:06.741279    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:07.272088 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.272024    4912 generic.go:356] "Generic (PLEG): container finished" podID="00484af4-8fb8-42d0-918e-597f661cef06" containerID="b26675e4e9ca8cd54635310a88bff3340dea29a07e3f64d3f721df8b81dcfe0f" exitCode=60
Nov 02 01:45:07.272295 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.272116    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq" event={"ID":"00484af4-8fb8-42d0-918e-597f661cef06","Type":"ContainerDied","Data":"b26675e4e9ca8cd54635310a88bff3340dea29a07e3f64d3f721df8b81dcfe0f"}
Nov 02 01:45:07.272399 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.272381    4912 scope.go:122] "RemoveContainer" containerID="b26675e4e9ca8cd54635310a88bff3340dea29a07e3f64d3f721df8b81dcfe0f"
Nov 02 01:45:07.273890 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.273866    4912 generic.go:356] "Generic (PLEG): container finished" podID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerID="b052f45c28144ffd61238cbdc40b308d1e116230b44cff85df55a606a89079e9" exitCode=60
Nov 02 01:45:07.274012 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.273948    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" event={"ID":"1a8f3dae-1842-4851-abf2-593fd1c81f3a","Type":"ContainerDied","Data":"b052f45c28144ffd61238cbdc40b308d1e116230b44cff85df55a606a89079e9"}
Nov 02 01:45:07.274289 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.274259    4912 scope.go:122] "RemoveContainer" containerID="b052f45c28144ffd61238cbdc40b308d1e116230b44cff85df55a606a89079e9"
Nov 02 01:45:07.276694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.276671    4912 generic.go:356] "Generic (PLEG): container finished" podID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerID="bc2f78f75fffec4cba349c90fc57289c0510042a09413b40012331d5f7bcca72" exitCode=60
Nov 02 01:45:07.277141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.277000    4912 scope.go:122] "RemoveContainer" containerID="bc2f78f75fffec4cba349c90fc57289c0510042a09413b40012331d5f7bcca72"
Nov 02 01:45:07.277287 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.277265    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" event={"ID":"3e67d151-dd2a-4191-8e5d-797bf2e9403f","Type":"ContainerDied","Data":"bc2f78f75fffec4cba349c90fc57289c0510042a09413b40012331d5f7bcca72"}
Nov 02 01:45:07.279124 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.279104    4912 generic.go:356] "Generic (PLEG): container finished" podID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerID="4f7a141f073ccf2e04d01f8d7c905ba62b64f87e0ed205a106ce40ac5a421994" exitCode=60
Nov 02 01:45:07.279201 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.279154    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v" event={"ID":"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071","Type":"ContainerDied","Data":"4f7a141f073ccf2e04d01f8d7c905ba62b64f87e0ed205a106ce40ac5a421994"}
Nov 02 01:45:07.279347 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:07.279336    4912 scope.go:122] "RemoveContainer" containerID="4f7a141f073ccf2e04d01f8d7c905ba62b64f87e0ed205a106ce40ac5a421994"
Nov 02 01:45:08.289672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.289640    4912 generic.go:356] "Generic (PLEG): container finished" podID="00484af4-8fb8-42d0-918e-597f661cef06" containerID="4b4f856b9972ef518d32f8bf461d72e95de81773d69fb376800d06fa338a7577" exitCode=60
Nov 02 01:45:08.290079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.289676    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq" event={"ID":"00484af4-8fb8-42d0-918e-597f661cef06","Type":"ContainerDied","Data":"4b4f856b9972ef518d32f8bf461d72e95de81773d69fb376800d06fa338a7577"}
Nov 02 01:45:08.290079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.289775    4912 scope.go:122] "RemoveContainer" containerID="b26675e4e9ca8cd54635310a88bff3340dea29a07e3f64d3f721df8b81dcfe0f"
Nov 02 01:45:08.290079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.290006    4912 scope.go:122] "RemoveContainer" containerID="4b4f856b9972ef518d32f8bf461d72e95de81773d69fb376800d06fa338a7577"
Nov 02 01:45:08.290322 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:08.290301    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq_odh-ai-gateway-infra(00484af4-8fb8-42d0-918e-597f661cef06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq" podUID="00484af4-8fb8-42d0-918e-597f661cef06"
Nov 02 01:45:08.291901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.291880    4912 generic.go:356] "Generic (PLEG): container finished" podID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerID="6d94aa283fb3b19253f5adeca67f1548d0cf851e65fcc287f94229478024207b" exitCode=60
Nov 02 01:45:08.292034 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.292010    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" event={"ID":"1a8f3dae-1842-4851-abf2-593fd1c81f3a","Type":"ContainerDied","Data":"6d94aa283fb3b19253f5adeca67f1548d0cf851e65fcc287f94229478024207b"}
Nov 02 01:45:08.292250 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.292220    4912 scope.go:122] "RemoveContainer" containerID="6d94aa283fb3b19253f5adeca67f1548d0cf851e65fcc287f94229478024207b"
Nov 02 01:45:08.292465 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:08.292441    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd_odh-ai-gateway-infra(1a8f3dae-1842-4851-abf2-593fd1c81f3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a"
Nov 02 01:45:08.294537 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.294503    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" event={"ID":"3e67d151-dd2a-4191-8e5d-797bf2e9403f","Type":"ContainerStarted","Data":"542eacf3c05188e08db3774ab08a35f3de0b3c6a41bfccafb77285316d4e3b28"}
Nov 02 01:45:08.296682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.296659    4912 generic.go:356] "Generic (PLEG): container finished" podID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerID="99a484636b36c0fabc06831fa6c16ff83852e5b5b9b302222da245baa2a2b4d8" exitCode=60
Nov 02 01:45:08.296772 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.296726    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v" event={"ID":"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071","Type":"ContainerDied","Data":"99a484636b36c0fabc06831fa6c16ff83852e5b5b9b302222da245baa2a2b4d8"}
Nov 02 01:45:08.296996 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.296979    4912 scope.go:122] "RemoveContainer" containerID="99a484636b36c0fabc06831fa6c16ff83852e5b5b9b302222da245baa2a2b4d8"
Nov 02 01:45:08.297258 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:08.297237    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v_odh-ai-gateway-infra(94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071"
Nov 02 01:45:08.471002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.470980    4912 scope.go:122] "RemoveContainer" containerID="b052f45c28144ffd61238cbdc40b308d1e116230b44cff85df55a606a89079e9"
Nov 02 01:45:08.488212 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.488194    4912 scope.go:122] "RemoveContainer" containerID="4f7a141f073ccf2e04d01f8d7c905ba62b64f87e0ed205a106ce40ac5a421994"
Nov 02 01:45:08.754523 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:08.754412    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" podStartSLOduration=4.015355108 podStartE2EDuration="8.754393113s" totalImagesPullingTime="4.739038005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:08.750327837 +0000 UTC m=+801.655793195" watchObservedRunningTime="2026-11-02 01:45:08.754393113 +0000 UTC m=+801.659858471"
Nov 02 01:45:09.304739 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:09.304700    4912 generic.go:356] "Generic (PLEG): container finished" podID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerID="542eacf3c05188e08db3774ab08a35f3de0b3c6a41bfccafb77285316d4e3b28" exitCode=60
Nov 02 01:45:09.305254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:09.304782    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" event={"ID":"3e67d151-dd2a-4191-8e5d-797bf2e9403f","Type":"ContainerDied","Data":"542eacf3c05188e08db3774ab08a35f3de0b3c6a41bfccafb77285316d4e3b28"}
Nov 02 01:45:09.305254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:09.304848    4912 scope.go:122] "RemoveContainer" containerID="bc2f78f75fffec4cba349c90fc57289c0510042a09413b40012331d5f7bcca72"
Nov 02 01:45:09.305254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:09.305019    4912 scope.go:122] "RemoveContainer" containerID="542eacf3c05188e08db3774ab08a35f3de0b3c6a41bfccafb77285316d4e3b28"
Nov 02 01:45:09.305447 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:09.305304    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd_odh-ai-gateway-infra(3e67d151-dd2a-4191-8e5d-797bf2e9403f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f"
Nov 02 01:45:09.306829 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:09.306806    4912 scope.go:122] "RemoveContainer" containerID="99a484636b36c0fabc06831fa6c16ff83852e5b5b9b302222da245baa2a2b4d8"
Nov 02 01:45:09.307089 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:09.307046    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v_odh-ai-gateway-infra(94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071"
Nov 02 01:45:09.308590 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:09.308574    4912 scope.go:122] "RemoveContainer" containerID="4b4f856b9972ef518d32f8bf461d72e95de81773d69fb376800d06fa338a7577"
Nov 02 01:45:09.308849 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:09.308822    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq_odh-ai-gateway-infra(00484af4-8fb8-42d0-918e-597f661cef06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq" podUID="00484af4-8fb8-42d0-918e-597f661cef06"
Nov 02 01:45:09.310051 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:09.310033    4912 scope.go:122] "RemoveContainer" containerID="6d94aa283fb3b19253f5adeca67f1548d0cf851e65fcc287f94229478024207b"
Nov 02 01:45:09.310263 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:09.310245    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd_odh-ai-gateway-infra(1a8f3dae-1842-4851-abf2-593fd1c81f3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a"
Nov 02 01:45:09.987959 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:09.987930    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:10.319933 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:10.319860    4912 scope.go:122] "RemoveContainer" containerID="542eacf3c05188e08db3774ab08a35f3de0b3c6a41bfccafb77285316d4e3b28"
Nov 02 01:45:10.320460 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:10.320264    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd_odh-ai-gateway-infra(3e67d151-dd2a-4191-8e5d-797bf2e9403f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f"
Nov 02 01:45:10.607890 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:10.607783    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:10.788702 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:10.788667    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:12.138281 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:12.138239    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:45:15.572464 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:15.572424    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:18.935046 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:18.935004    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:19.218529 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:19.218457    4912 scope.go:122] "RemoveContainer" containerID="4b4f856b9972ef518d32f8bf461d72e95de81773d69fb376800d06fa338a7577"
Nov 02 01:45:20.218763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:20.218731    4912 scope.go:122] "RemoveContainer" containerID="542eacf3c05188e08db3774ab08a35f3de0b3c6a41bfccafb77285316d4e3b28"
Nov 02 01:45:20.389700 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:20.389664    4912 generic.go:356] "Generic (PLEG): container finished" podID="00484af4-8fb8-42d0-918e-597f661cef06" containerID="57380727f5ae57eb374332d3a2d2463a7d396a13dbee6256c31e071aec3e8438" exitCode=60
Nov 02 01:45:20.389876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:20.389733    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq" event={"ID":"00484af4-8fb8-42d0-918e-597f661cef06","Type":"ContainerDied","Data":"57380727f5ae57eb374332d3a2d2463a7d396a13dbee6256c31e071aec3e8438"}
Nov 02 01:45:20.389876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:20.389794    4912 scope.go:122] "RemoveContainer" containerID="4b4f856b9972ef518d32f8bf461d72e95de81773d69fb376800d06fa338a7577"
Nov 02 01:45:20.390165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:20.390151    4912 scope.go:122] "RemoveContainer" containerID="57380727f5ae57eb374332d3a2d2463a7d396a13dbee6256c31e071aec3e8438"
Nov 02 01:45:20.390459 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:20.390439    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq_odh-ai-gateway-infra(00484af4-8fb8-42d0-918e-597f661cef06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq" podUID="00484af4-8fb8-42d0-918e-597f661cef06"
Nov 02 01:45:20.459043 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:20.459009    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:20.775031 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:20.774951    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:21.042270 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.042175    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:21.220097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.218973    4912 scope.go:122] "RemoveContainer" containerID="99a484636b36c0fabc06831fa6c16ff83852e5b5b9b302222da245baa2a2b4d8"
Nov 02 01:45:21.224843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.224816    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:21.400431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.400403    4912 generic.go:356] "Generic (PLEG): container finished" podID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerID="89afc8cecb151a6ac1f61ecbc66ba6f5b6dd2998934e135ab8107fa2be41656d" exitCode=60
Nov 02 01:45:21.400604 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.400456    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" event={"ID":"3e67d151-dd2a-4191-8e5d-797bf2e9403f","Type":"ContainerDied","Data":"89afc8cecb151a6ac1f61ecbc66ba6f5b6dd2998934e135ab8107fa2be41656d"}
Nov 02 01:45:21.400604 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.400505    4912 scope.go:122] "RemoveContainer" containerID="542eacf3c05188e08db3774ab08a35f3de0b3c6a41bfccafb77285316d4e3b28"
Nov 02 01:45:21.400866 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.400852    4912 scope.go:122] "RemoveContainer" containerID="89afc8cecb151a6ac1f61ecbc66ba6f5b6dd2998934e135ab8107fa2be41656d"
Nov 02 01:45:21.401182 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:21.401162    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd_odh-ai-gateway-infra(3e67d151-dd2a-4191-8e5d-797bf2e9403f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f"
Nov 02 01:45:21.516024 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.515991    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"]
Nov 02 01:45:21.681503 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.681475    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:21.797959 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.797921    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00484af4-8fb8-42d0-918e-597f661cef06-kube-api-access-jzzgh\" (UniqueName: \"kubernetes.io/projected/00484af4-8fb8-42d0-918e-597f661cef06-kube-api-access-jzzgh\") pod \"00484af4-8fb8-42d0-918e-597f661cef06\" (UID: \"00484af4-8fb8-42d0-918e-597f661cef06\") "
Nov 02 01:45:21.798184 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.797989    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00484af4-8fb8-42d0-918e-597f661cef06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00484af4-8fb8-42d0-918e-597f661cef06-maas-api-service-ca\") pod \"00484af4-8fb8-42d0-918e-597f661cef06\" (UID: \"00484af4-8fb8-42d0-918e-597f661cef06\") "
Nov 02 01:45:21.798824 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.798793    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00484af4-8fb8-42d0-918e-597f661cef06-maas-api-service-ca" pod "00484af4-8fb8-42d0-918e-597f661cef06" (UID: "00484af4-8fb8-42d0-918e-597f661cef06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:21.801897 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.801712    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00484af4-8fb8-42d0-918e-597f661cef06-kube-api-access-jzzgh" pod "00484af4-8fb8-42d0-918e-597f661cef06" (UID: "00484af4-8fb8-42d0-918e-597f661cef06"). InnerVolumeSpecName "kube-api-access-jzzgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:21.899204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.899166    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzzgh\" (UniqueName: \"kubernetes.io/projected/00484af4-8fb8-42d0-918e-597f661cef06-kube-api-access-jzzgh\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:21.899204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:21.899194    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00484af4-8fb8-42d0-918e-597f661cef06-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:22.217979 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.217944    4912 scope.go:122] "RemoveContainer" containerID="6d94aa283fb3b19253f5adeca67f1548d0cf851e65fcc287f94229478024207b"
Nov 02 01:45:22.408078 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.408035    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"
Nov 02 01:45:22.408558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.408083    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq" event={"ID":"00484af4-8fb8-42d0-918e-597f661cef06","Type":"ContainerDied","Data":"aac791660fbf1222cc8ab4139800c937af4e2010353cdbc73f0f105067a9f9a1"}
Nov 02 01:45:22.408558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.408150    4912 scope.go:122] "RemoveContainer" containerID="57380727f5ae57eb374332d3a2d2463a7d396a13dbee6256c31e071aec3e8438"
Nov 02 01:45:22.412515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.412483    4912 generic.go:356] "Generic (PLEG): container finished" podID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerID="cdb7b2a80c96397dc7a81dd5a516fee2906a596203d8f6adcea5735dd6e86953" exitCode=60
Nov 02 01:45:22.412646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.412534    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v" event={"ID":"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071","Type":"ContainerDied","Data":"cdb7b2a80c96397dc7a81dd5a516fee2906a596203d8f6adcea5735dd6e86953"}
Nov 02 01:45:22.412896 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.412879    4912 scope.go:122] "RemoveContainer" containerID="cdb7b2a80c96397dc7a81dd5a516fee2906a596203d8f6adcea5735dd6e86953"
Nov 02 01:45:22.413155 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:22.413137    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v_odh-ai-gateway-infra(94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071"
Nov 02 01:45:22.418762 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.418742    4912 scope.go:122] "RemoveContainer" containerID="99a484636b36c0fabc06831fa6c16ff83852e5b5b9b302222da245baa2a2b4d8"
Nov 02 01:45:22.529601 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.529565    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"]
Nov 02 01:45:22.572367 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.572335    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"]
Nov 02 01:45:22.576900 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.576864    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893065-9k7fq"]
Nov 02 01:45:22.690165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.690136    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:22.809340 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.809310    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e67d151-dd2a-4191-8e5d-797bf2e9403f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e67d151-dd2a-4191-8e5d-797bf2e9403f-maas-api-service-ca\") pod \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\" (UID: \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\") "
Nov 02 01:45:22.809518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.809416    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e67d151-dd2a-4191-8e5d-797bf2e9403f-kube-api-access-n8f4t\" (UniqueName: \"kubernetes.io/projected/3e67d151-dd2a-4191-8e5d-797bf2e9403f-kube-api-access-n8f4t\") pod \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\" (UID: \"3e67d151-dd2a-4191-8e5d-797bf2e9403f\") "
Nov 02 01:45:22.809668 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.809644    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e67d151-dd2a-4191-8e5d-797bf2e9403f-maas-api-service-ca" pod "3e67d151-dd2a-4191-8e5d-797bf2e9403f" (UID: "3e67d151-dd2a-4191-8e5d-797bf2e9403f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:22.811820 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.811793    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e67d151-dd2a-4191-8e5d-797bf2e9403f-kube-api-access-n8f4t" pod "3e67d151-dd2a-4191-8e5d-797bf2e9403f" (UID: "3e67d151-dd2a-4191-8e5d-797bf2e9403f"). InnerVolumeSpecName "kube-api-access-n8f4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:22.910598 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.910567    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e67d151-dd2a-4191-8e5d-797bf2e9403f-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:22.910598 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:22.910596    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8f4t\" (UniqueName: \"kubernetes.io/projected/3e67d151-dd2a-4191-8e5d-797bf2e9403f-kube-api-access-n8f4t\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:23.225755 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.225726    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00484af4-8fb8-42d0-918e-597f661cef06" path="/var/lib/kubelet/pods/00484af4-8fb8-42d0-918e-597f661cef06/volumes"
Nov 02 01:45:23.425734 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.425633    4912 generic.go:356] "Generic (PLEG): container finished" podID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerID="e0ec72a1f75610337aedc546213bbf666c6cebb194ccb530f20bcf982e577e6c" exitCode=60
Nov 02 01:45:23.425734 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.425702    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" event={"ID":"1a8f3dae-1842-4851-abf2-593fd1c81f3a","Type":"ContainerDied","Data":"e0ec72a1f75610337aedc546213bbf666c6cebb194ccb530f20bcf982e577e6c"}
Nov 02 01:45:23.426298 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.425759    4912 scope.go:122] "RemoveContainer" containerID="6d94aa283fb3b19253f5adeca67f1548d0cf851e65fcc287f94229478024207b"
Nov 02 01:45:23.426298 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.426138    4912 scope.go:122] "RemoveContainer" containerID="e0ec72a1f75610337aedc546213bbf666c6cebb194ccb530f20bcf982e577e6c"
Nov 02 01:45:23.426454 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:23.426413    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd_odh-ai-gateway-infra(1a8f3dae-1842-4851-abf2-593fd1c81f3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a"
Nov 02 01:45:23.427422 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.427400    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd" event={"ID":"3e67d151-dd2a-4191-8e5d-797bf2e9403f","Type":"ContainerDied","Data":"bf4228fa8462968e866dc1b0e66db4110a83677c4f01d513a046ef133210ef99"}
Nov 02 01:45:23.427422 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.427432    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"
Nov 02 01:45:23.440883 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.440866    4912 scope.go:122] "RemoveContainer" containerID="89afc8cecb151a6ac1f61ecbc66ba6f5b6dd2998934e135ab8107fa2be41656d"
Nov 02 01:45:23.656767 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.656722    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"]
Nov 02 01:45:23.661506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.661479    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893065-hnkcd"]
Nov 02 01:45:23.699647 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.699570    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"]
Nov 02 01:45:23.848327 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.848300    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:23.921010 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.920983    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-kube-api-access-xfnv7\" (UniqueName: \"kubernetes.io/projected/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-kube-api-access-xfnv7\") pod \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\" (UID: \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\") "
Nov 02 01:45:23.921213 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.921030    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-maas-api-service-ca\") pod \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\" (UID: \"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071\") "
Nov 02 01:45:23.921397 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.921372    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-maas-api-service-ca" pod "94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" (UID: "94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:23.923438 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:23.923406    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-kube-api-access-xfnv7" pod "94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" (UID: "94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071"). InnerVolumeSpecName "kube-api-access-xfnv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:24.022905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.022823    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnv7\" (UniqueName: \"kubernetes.io/projected/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-kube-api-access-xfnv7\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:24.022905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.022847    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:24.438736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.438703    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"
Nov 02 01:45:24.439204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.438702    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v" event={"ID":"94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071","Type":"ContainerDied","Data":"6455a0f79d76e4c7b8ed6294cc3e3d3a6c5bb9acf103b70be65d4fb9e74a3493"}
Nov 02 01:45:24.439204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.438835    4912 scope.go:122] "RemoveContainer" containerID="cdb7b2a80c96397dc7a81dd5a516fee2906a596203d8f6adcea5735dd6e86953"
Nov 02 01:45:24.595082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.594381    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"]
Nov 02 01:45:24.619083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.618695    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"]
Nov 02 01:45:24.633082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.624907    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893065-76z6v"]
Nov 02 01:45:24.808287 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.808250    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:24.946332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.946282    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8f3dae-1842-4851-abf2-593fd1c81f3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f3dae-1842-4851-abf2-593fd1c81f3a-maas-api-service-ca\") pod \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\" (UID: \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\") "
Nov 02 01:45:24.946505 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.946368    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8f3dae-1842-4851-abf2-593fd1c81f3a-kube-api-access-nvdp7\" (UniqueName: \"kubernetes.io/projected/1a8f3dae-1842-4851-abf2-593fd1c81f3a-kube-api-access-nvdp7\") pod \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\" (UID: \"1a8f3dae-1842-4851-abf2-593fd1c81f3a\") "
Nov 02 01:45:24.956921 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.956842    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8f3dae-1842-4851-abf2-593fd1c81f3a-maas-api-service-ca" pod "1a8f3dae-1842-4851-abf2-593fd1c81f3a" (UID: "1a8f3dae-1842-4851-abf2-593fd1c81f3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:24.963887 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:24.963838    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8f3dae-1842-4851-abf2-593fd1c81f3a-kube-api-access-nvdp7" pod "1a8f3dae-1842-4851-abf2-593fd1c81f3a" (UID: "1a8f3dae-1842-4851-abf2-593fd1c81f3a"). InnerVolumeSpecName "kube-api-access-nvdp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:25.048370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.048334    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f3dae-1842-4851-abf2-593fd1c81f3a-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:25.048370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.048367    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvdp7\" (UniqueName: \"kubernetes.io/projected/1a8f3dae-1842-4851-abf2-593fd1c81f3a-kube-api-access-nvdp7\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:25.227597 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.227511    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" path="/var/lib/kubelet/pods/3e67d151-dd2a-4191-8e5d-797bf2e9403f/volumes"
Nov 02 01:45:25.227943 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.227929    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" path="/var/lib/kubelet/pods/94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071/volumes"
Nov 02 01:45:25.447734 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.447706    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"
Nov 02 01:45:25.448220 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.447751    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd" event={"ID":"1a8f3dae-1842-4851-abf2-593fd1c81f3a","Type":"ContainerDied","Data":"a06e07856bdab8a93a3f65e0d097921da51910bd3a2241c63e8206462845b767"}
Nov 02 01:45:25.448220 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.447815    4912 scope.go:122] "RemoveContainer" containerID="e0ec72a1f75610337aedc546213bbf666c6cebb194ccb530f20bcf982e577e6c"
Nov 02 01:45:25.611546 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.611469    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"]
Nov 02 01:45:25.616484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.616457    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893065-9ftxd"]
Nov 02 01:45:25.725415 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:25.725382    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:26.090446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:26.090404    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:26.318873 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:26.318840    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:26.548004 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:26.547968    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:26.760847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:26.760806    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:27.224733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:27.224699    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" path="/var/lib/kubelet/pods/1a8f3dae-1842-4851-abf2-593fd1c81f3a/volumes"
Nov 02 01:45:30.862582 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:30.862542    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:31.601628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:31.601597    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:32.185907 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:32.185862    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:32.411474 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:32.411435    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:36.052718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:36.052684    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:36.811546 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:36.811506    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:40.863686 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:40.863649    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:42.143492 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:45:42.143448    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:45:42.195792 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:42.195762    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:45.561664 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:45.561628    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:49.363410 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.363361    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"]
Nov 02 01:45:49.364265 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364234    4912 cpu_manager.go:414] "containerMap: removing container" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerName="cleanup"
Nov 02 01:45:49.364265 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364265    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364301    4912 cpu_manager.go:414] "containerMap: removing container" podUID="00484af4-8fb8-42d0-918e-597f661cef06" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364317    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00484af4-8fb8-42d0-918e-597f661cef06" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364327    4912 cpu_manager.go:414] "containerMap: removing container" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364334    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364345    4912 cpu_manager.go:414] "containerMap: removing container" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364352    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364368    4912 cpu_manager.go:414] "containerMap: removing container" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364374    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364384    4912 cpu_manager.go:414] "containerMap: removing container" podUID="00484af4-8fb8-42d0-918e-597f661cef06" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364390    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00484af4-8fb8-42d0-918e-597f661cef06" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364419    4912 cpu_manager.go:414] "containerMap: removing container" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerName="cleanup"
Nov 02 01:45:49.364424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364426    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364442    4912 cpu_manager.go:414] "containerMap: removing container" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364449    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364458    4912 cpu_manager.go:414] "containerMap: removing container" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364464    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364476    4912 cpu_manager.go:414] "containerMap: removing container" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364482    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364495    4912 cpu_manager.go:414] "containerMap: removing container" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364503    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364511    4912 cpu_manager.go:414] "containerMap: removing container" podUID="00484af4-8fb8-42d0-918e-597f661cef06" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364517    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00484af4-8fb8-42d0-918e-597f661cef06" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364643    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364655    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="00484af4-8fb8-42d0-918e-597f661cef06" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364667    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364676    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364684    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364694    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364702    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="00484af4-8fb8-42d0-918e-597f661cef06" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364711    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="00484af4-8fb8-42d0-918e-597f661cef06" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364720    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerName="cleanup"
Nov 02 01:45:49.364773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.364729    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e67d151-dd2a-4191-8e5d-797bf2e9403f" containerName="cleanup"
Nov 02 01:45:49.373153 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.373127    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.417476 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.417447    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-dshm\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.417674 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.417486    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-tmp-dir\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.417674 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.417508    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlqt\" (UniqueName: \"kubernetes.io/projected/24e41a89-3a5d-4bb8-993c-8db65b10e224-kube-api-access-rmlqt\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.417798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.417705    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-home\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.417798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.417754    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24e41a89-3a5d-4bb8-993c-8db65b10e224-tls-certs\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.417859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.417811    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-model-cache\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.424305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.424276    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"]
Nov 02 01:45:49.425767 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.425729    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0ebdb3\"/\"kube-root-ca.crt\""
Nov 02 01:45:49.425767 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.425743    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ebdb3\"/\"simulator-w6-0ebdb3-kserve-self-signed-certs\""
Nov 02 01:45:49.425917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.425769    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0ebdb3\"/\"openshift-service-ca.crt\""
Nov 02 01:45:49.425917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.425872    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ebdb3\"/\"default-dockercfg-gtrzx\""
Nov 02 01:45:49.491135 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.491097    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:49.519790 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.519754    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-home\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.519978 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.519805    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24e41a89-3a5d-4bb8-993c-8db65b10e224-tls-certs\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.519978 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.519841    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-model-cache\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.520129 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.520003    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-dshm\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.520129 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.520046    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-tmp-dir\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.520129 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.520092    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlqt\" (UniqueName: \"kubernetes.io/projected/24e41a89-3a5d-4bb8-993c-8db65b10e224-kube-api-access-rmlqt\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.520289 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.520265    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-home\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.520346 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.520271    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-model-cache\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.520592 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.520570    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-tmp-dir\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.522452 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.522431    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-dshm\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.522639 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.522621    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24e41a89-3a5d-4bb8-993c-8db65b10e224-tls-certs\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.582178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.582147    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlqt\" (UniqueName: \"kubernetes.io/projected/24e41a89-3a5d-4bb8-993c-8db65b10e224-kube-api-access-rmlqt\") pod \"simulator-w6-0ebdb3-kserve-985cc5f47-j9slf\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") " pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:49.687786 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:49.687748    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:50.075771 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:45:50.075739    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24e41a89_3a5d_4bb8_993c_8db65b10e224.slice/crio-0012ed5fc6a8e526f391e779825b735a725848de4e5ea2dd17d298ad664a896b WatchSource:0}: Error finding container 0012ed5fc6a8e526f391e779825b735a725848de4e5ea2dd17d298ad664a896b: Status 404 returned error can't find the container with id 0012ed5fc6a8e526f391e779825b735a725848de4e5ea2dd17d298ad664a896b
Nov 02 01:45:50.076732 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:50.076706    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"]
Nov 02 01:45:50.449808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:50.449771    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:50.639695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:50.639599    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" event={"ID":"24e41a89-3a5d-4bb8-993c-8db65b10e224","Type":"ContainerStarted","Data":"2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e"}
Nov 02 01:45:50.639695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:50.639660    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" event={"ID":"24e41a89-3a5d-4bb8-993c-8db65b10e224","Type":"ContainerStarted","Data":"0012ed5fc6a8e526f391e779825b735a725848de4e5ea2dd17d298ad664a896b"}
Nov 02 01:45:50.688748 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:50.688714    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:50.760184 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:50.760118    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podStartSLOduration=1.7600953559999999 podStartE2EDuration="1.760095356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:50.756005889 +0000 UTC m=+843.661471253" watchObservedRunningTime="2026-11-02 01:45:50.760095356 +0000 UTC m=+843.665560721"
Nov 02 01:45:55.211174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:55.211143    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:55.430694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:55.430642    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:56.464103 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:56.464052    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:59.620344 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:59.620138    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:45:59.687882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:59.687851    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:59.702665 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:59.702634    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:45:59.715493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:45:59.715465    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:46:00.049695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:00.049597    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:04.854636 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.854576    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:04.882482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.882449    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"]
Nov 02 01:46:04.883126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.883112    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8f3dae-1842-4851-abf2-593fd1c81f3a" containerName="cleanup"
Nov 02 01:46:04.883176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.883130    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f7b8bc-ead3-4eb9-a410-6bcc3aa6b071" containerName="cleanup"
Nov 02 01:46:04.886445 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.886431    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:04.989141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.989105    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"openshift-service-ca.crt\""
Nov 02 01:46:04.989347 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.989184    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"kube-root-ca.crt\""
Nov 02 01:46:04.989347 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.989250    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"default-dockercfg-qlw9w\""
Nov 02 01:46:04.989347 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.989323    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"simulator-w3-061fab-kserve-self-signed-certs\""
Nov 02 01:46:04.990974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.990953    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-model-cache\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:04.991147 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.991005    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-home\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:04.991221 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.991201    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2014bd97-f76a-4090-b50e-45448ff71bf9-tls-certs\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:04.991271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.991252    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-dshm\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:04.991332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.991316    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-tmp-dir\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:04.991411 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.991395    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64l9l\" (UniqueName: \"kubernetes.io/projected/2014bd97-f76a-4090-b50e-45448ff71bf9-kube-api-access-64l9l\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:04.992419 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:04.992399    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"]
Nov 02 01:46:05.031437 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.031404    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:05.092691 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.092628    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2014bd97-f76a-4090-b50e-45448ff71bf9-tls-certs\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.092919 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.092694    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-dshm\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.092919 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.092740    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-tmp-dir\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.092919 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.092789    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64l9l\" (UniqueName: \"kubernetes.io/projected/2014bd97-f76a-4090-b50e-45448ff71bf9-kube-api-access-64l9l\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.092919 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.092881    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-model-cache\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.093198 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.092922    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-home\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.093493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.093469    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-model-cache\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.093646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.093507    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-home\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.093715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.093671    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-tmp-dir\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.095560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.095528    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-dshm\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.099962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.099936    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2014bd97-f76a-4090-b50e-45448ff71bf9-tls-certs\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.154031 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.153993    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64l9l\" (UniqueName: \"kubernetes.io/projected/2014bd97-f76a-4090-b50e-45448ff71bf9-kube-api-access-64l9l\") pod \"simulator-w3-061fab-kserve-797df4ff5b-wbqrs\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") " pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.199590 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.199560    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:05.580248 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.580222    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"]
Nov 02 01:46:05.580971 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:46:05.580944    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2014bd97_f76a_4090_b50e_45448ff71bf9.slice/crio-7af7398d7b87f8fd32b8021e85c62cecc5a2c7ed9100b435fd69716ea2921ede WatchSource:0}: Error finding container 7af7398d7b87f8fd32b8021e85c62cecc5a2c7ed9100b435fd69716ea2921ede: Status 404 returned error can't find the container with id 7af7398d7b87f8fd32b8021e85c62cecc5a2c7ed9100b435fd69716ea2921ede
Nov 02 01:46:05.746356 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.746318    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" event={"ID":"2014bd97-f76a-4090-b50e-45448ff71bf9","Type":"ContainerStarted","Data":"aef0f75a57a27a157dddad1f30f36fc32166324aa3ded6a39a64f6c598fc9f74"}
Nov 02 01:46:05.746552 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.746369    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" event={"ID":"2014bd97-f76a-4090-b50e-45448ff71bf9","Type":"ContainerStarted","Data":"7af7398d7b87f8fd32b8021e85c62cecc5a2c7ed9100b435fd69716ea2921ede"}
Nov 02 01:46:05.868134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:05.868078    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podStartSLOduration=1.8680494429999999 podStartE2EDuration="1.868049443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:05.864613774 +0000 UTC m=+858.770079134" watchObservedRunningTime="2026-11-02 01:46:05.868049443 +0000 UTC m=+858.773514799"
Nov 02 01:46:06.200096 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:06.199988    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:09.611252 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:09.611219    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:09.785496 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:09.785462    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:12.138276 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:46:12.138234    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:46:14.426308 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:14.426271    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:15.200141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:15.200108    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:15.213486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:15.213460    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:15.828491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:15.828465    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:46:19.146811 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:19.146775    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:23.963684 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:23.963645    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:28.420104 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:28.420050    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:28.742843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:28.742759    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:33.454497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:33.454457    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:38.243280 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:38.243239    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:42.138787 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:46:42.138747    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:46:42.549276 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:42.549192    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:44.654769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:44.654737    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:47.465024 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.464997    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:46:47.472821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.472794    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:46:47.511718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.511693    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:46:47.518303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.518280    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:46:47.650271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.650239    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"]
Nov 02 01:46:47.654482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.654459    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.713412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.713379    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"]
Nov 02 01:46:47.743178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.743095    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dbe849\"/\"kube-root-ca.crt\""
Nov 02 01:46:47.757889 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.757862    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dbe849\"/\"openshift-service-ca.crt\""
Nov 02 01:46:47.758092 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.757969    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dbe849\"/\"default-dockercfg-wkvbk\""
Nov 02 01:46:47.758092 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.758052    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dbe849\"/\"premium-w1-dbe849-kserve-self-signed-certs\""
Nov 02 01:46:47.770660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.770630    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-home\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.770832 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.770683    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-tmp-dir\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.770906 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.770832    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rkf6\" (UniqueName: \"kubernetes.io/projected/569172aa-26cb-46aa-8a18-d46f3c7630c6-kube-api-access-5rkf6\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.770962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.770938    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-model-cache\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.771037 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.771014    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569172aa-26cb-46aa-8a18-d46f3c7630c6-tls-certs\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.771119 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.771048    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-dshm\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.779262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.779225    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:47.872610 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.872570    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-tmp-dir\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.872819 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.872631    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rkf6\" (UniqueName: \"kubernetes.io/projected/569172aa-26cb-46aa-8a18-d46f3c7630c6-kube-api-access-5rkf6\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.872819 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.872714    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-model-cache\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.872819 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.872777    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569172aa-26cb-46aa-8a18-d46f3c7630c6-tls-certs\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.872819 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.872808    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-dshm\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.873108 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.872934    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-home\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.873108 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.873012    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-tmp-dir\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.873334 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.873309    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-home\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.873997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.873952    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-model-cache\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.876590 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.876565    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569172aa-26cb-46aa-8a18-d46f3c7630c6-tls-certs\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.877808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.877782    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-dshm\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.933995 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.933964    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rkf6\" (UniqueName: \"kubernetes.io/projected/569172aa-26cb-46aa-8a18-d46f3c7630c6-kube-api-access-5rkf6\") pod \"premium-w1-dbe849-kserve-76c4b7cf6f-j4br2\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") " pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:47.956227 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.956196    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:47.967426 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:47.967395    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:48.367335 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:48.367305    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"]
Nov 02 01:46:48.367823 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:46:48.367796    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod569172aa_26cb_46aa_8a18_d46f3c7630c6.slice/crio-cb4c002a16e458317b2afd7056f03091c2abcd55fc6728bf58b2f87753737f0b WatchSource:0}: Error finding container cb4c002a16e458317b2afd7056f03091c2abcd55fc6728bf58b2f87753737f0b: Status 404 returned error can't find the container with id cb4c002a16e458317b2afd7056f03091c2abcd55fc6728bf58b2f87753737f0b
Nov 02 01:46:49.087085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:49.079781    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" event={"ID":"569172aa-26cb-46aa-8a18-d46f3c7630c6","Type":"ContainerStarted","Data":"67c2ec2b8ecc3b68dfe80d4d269477797b26089eda70923ca1dda36b04bba890"}
Nov 02 01:46:49.087085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:49.079996    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" event={"ID":"569172aa-26cb-46aa-8a18-d46f3c7630c6","Type":"ContainerStarted","Data":"cb4c002a16e458317b2afd7056f03091c2abcd55fc6728bf58b2f87753737f0b"}
Nov 02 01:46:49.221812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:49.221763    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podStartSLOduration=2.221749826 podStartE2EDuration="2.221749826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:49.22069971 +0000 UTC m=+902.126165071" watchObservedRunningTime="2026-11-02 01:46:49.221749826 +0000 UTC m=+902.127215183"
Nov 02 01:46:49.968354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:49.968326    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:52.757596 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:52.757561    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:57.607473 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:57.607434    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:46:57.967733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:57.967702    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:57.980050 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:57.980026    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:46:58.156857 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:46:58.156828    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:47:02.344112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:02.344051    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:03.305486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:03.305447    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:03.482232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:03.482196    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:04.729433 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:04.729388    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:07.148603 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:07.148573    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:11.865661 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:11.865612    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:12.137236 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:47:12.137149    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:47:16.690875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:16.690836    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:19.684158 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.684120    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"]
Nov 02 01:47:19.688249 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.688223    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.757801 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.757772    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-tmp-dir\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.757961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.757828    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefb2f44-77e8-45cb-9ec6-017183892791-tls-certs\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.757961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.757910    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mppxm\" (UniqueName: \"kubernetes.io/projected/aefb2f44-77e8-45cb-9ec6-017183892791-kube-api-access-mppxm\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.758072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.758015    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-home\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.758072 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.758037    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-dshm\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.758174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.758152    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-model-cache\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.791067 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.791023    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ebdb3\"/\"premium-w6-0ebdb3-kserve-self-signed-certs\""
Nov 02 01:47:19.794486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.794466    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"]
Nov 02 01:47:19.810443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.810412    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:19.859824 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.859783    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-model-cache\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.859988 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.859925    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-tmp-dir\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.860046 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.859984    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefb2f44-77e8-45cb-9ec6-017183892791-tls-certs\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.860046 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.860018    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppxm\" (UniqueName: \"kubernetes.io/projected/aefb2f44-77e8-45cb-9ec6-017183892791-kube-api-access-mppxm\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.860239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.860126    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-home\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.860239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.860146    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-dshm\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.860504 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.860480    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-tmp-dir\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.860504 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.860496    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-model-cache\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.860851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.860807    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-home\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.862972 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.862933    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-dshm\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.864191 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.864168    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefb2f44-77e8-45cb-9ec6-017183892791-tls-certs\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:19.921477 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:19.921450    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppxm\" (UniqueName: \"kubernetes.io/projected/aefb2f44-77e8-45cb-9ec6-017183892791-kube-api-access-mppxm\") pod \"premium-w6-0ebdb3-kserve-784d7588fd-6l622\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") " pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:20.001134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:20.001020    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:20.376577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:20.376551    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"]
Nov 02 01:47:20.377292 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:47:20.377270    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaefb2f44_77e8_45cb_9ec6_017183892791.slice/crio-f2e0ff17ebab4e9f181152778a226961922877da3764dbee59e6ecc13b4acda2 WatchSource:0}: Error finding container f2e0ff17ebab4e9f181152778a226961922877da3764dbee59e6ecc13b4acda2: Status 404 returned error can't find the container with id f2e0ff17ebab4e9f181152778a226961922877da3764dbee59e6ecc13b4acda2
Nov 02 01:47:21.313639 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:21.313599    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" event={"ID":"aefb2f44-77e8-45cb-9ec6-017183892791","Type":"ContainerStarted","Data":"1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1"}
Nov 02 01:47:21.314027 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:21.313654    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" event={"ID":"aefb2f44-77e8-45cb-9ec6-017183892791","Type":"ContainerStarted","Data":"f2e0ff17ebab4e9f181152778a226961922877da3764dbee59e6ecc13b4acda2"}
Nov 02 01:47:21.430666 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:21.430619    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podStartSLOduration=2.430605974 podStartE2EDuration="2.430605974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:21.42911868 +0000 UTC m=+934.334584035" watchObservedRunningTime="2026-11-02 01:47:21.430605974 +0000 UTC m=+934.336071330"
Nov 02 01:47:21.433991 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:21.433966    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:22.002077 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:22.001143    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:26.136596 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:26.136554    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:30.002033 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:30.001944    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:30.015274 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:30.015232    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:30.392460 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:30.392430    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:47:30.946459 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:30.946429    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:35.115818 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:35.115777    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:35.655916 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:35.655881    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:40.500431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:40.500334    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:42.138188 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:47:42.138151    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:47:42.906297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:42.906258    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:45.362861 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:45.362824    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:49.119107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:49.119052    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:50.142287 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:50.142248    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:54.870913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:54.870865    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:47:59.645580 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:47:59.645528    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:04.464249 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:04.464196    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:09.268493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:09.267320    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:12.138162 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:48:12.138126    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:48:14.055677 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:14.055536    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:18.848773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:18.848739    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:21.613774 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.613737    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"]
Nov 02 01:48:21.617783 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.617761    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.619302 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.619276    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2457cd74-7gj89\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.619417 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.619405    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2457cd74-7gj89\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.619475 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.619435    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nck8l\" (UniqueName: \"kubernetes.io/projected/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-kube-api-access-nck8l\") pod \"maas-api-revoke-sub-2457cd74-7gj89\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.674627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.674591    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"]
Nov 02 01:48:21.725718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.720136    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:21.725718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.720333    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:48:21.725718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.720421    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2457cd74-7gj89\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.725718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.720558    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2457cd74-7gj89\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.725718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.720603    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nck8l\" (UniqueName: \"kubernetes.io/projected/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-kube-api-access-nck8l\") pod \"maas-api-revoke-sub-2457cd74-7gj89\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.725718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.721723    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2457cd74-7gj89\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.725718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.724303    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2457cd74-7gj89\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.782471 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.782437    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nck8l\" (UniqueName: \"kubernetes.io/projected/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-kube-api-access-nck8l\") pod \"maas-api-revoke-sub-2457cd74-7gj89\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:21.931355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:21.931321    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:22.309702 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:22.309664    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"]
Nov 02 01:48:22.310910 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:48:22.310881    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d9d7ca_3c71_49b8_babb_2d08e6e49221.slice/crio-0d9c1b61df78d4e6a48ad1645fc9f80c31326e3ab2125ebaaad00107d90e6d94 WatchSource:0}: Error finding container 0d9c1b61df78d4e6a48ad1645fc9f80c31326e3ab2125ebaaad00107d90e6d94: Status 404 returned error can't find the container with id 0d9c1b61df78d4e6a48ad1645fc9f80c31326e3ab2125ebaaad00107d90e6d94
Nov 02 01:48:22.786589 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:22.786557    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89" event={"ID":"b6d9d7ca-3c71-49b8-babb-2d08e6e49221","Type":"ContainerStarted","Data":"d7d3438b9f35ad6d7a124bad48bc2395cbae0013e70dcece55f6bf8789de2e54"}
Nov 02 01:48:22.786981 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:22.786604    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89" event={"ID":"b6d9d7ca-3c71-49b8-babb-2d08e6e49221","Type":"ContainerStarted","Data":"0d9c1b61df78d4e6a48ad1645fc9f80c31326e3ab2125ebaaad00107d90e6d94"}
Nov 02 01:48:22.939428 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:22.939374    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89" podStartSLOduration=1.9393552440000001 podStartE2EDuration="1.939355244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:22.937582525 +0000 UTC m=+995.843047883" watchObservedRunningTime="2026-11-02 01:48:22.939355244 +0000 UTC m=+995.844820600"
Nov 02 01:48:23.670579 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:23.670536    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:23.794673 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:23.794641    4912 generic.go:356] "Generic (PLEG): container finished" podID="b6d9d7ca-3c71-49b8-babb-2d08e6e49221" containerID="d7d3438b9f35ad6d7a124bad48bc2395cbae0013e70dcece55f6bf8789de2e54" exitCode=0
Nov 02 01:48:23.795097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:23.794719    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89" event={"ID":"b6d9d7ca-3c71-49b8-babb-2d08e6e49221","Type":"ContainerDied","Data":"d7d3438b9f35ad6d7a124bad48bc2395cbae0013e70dcece55f6bf8789de2e54"}
Nov 02 01:48:24.971892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:24.971866    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:25.064688 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.064596    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-openshift-ca\") pod \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") "
Nov 02 01:48:25.064867 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.064770    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-kube-api-access-nck8l\" (UniqueName: \"kubernetes.io/projected/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-kube-api-access-nck8l\") pod \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") "
Nov 02 01:48:25.064867 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.064809    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-xks-ca\") pod \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\" (UID: \"b6d9d7ca-3c71-49b8-babb-2d08e6e49221\") "
Nov 02 01:48:25.064987 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.064913    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-openshift-ca" pod "b6d9d7ca-3c71-49b8-babb-2d08e6e49221" (UID: "b6d9d7ca-3c71-49b8-babb-2d08e6e49221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:25.065265 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.065244    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:25.067073 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.067033    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-kube-api-access-nck8l" pod "b6d9d7ca-3c71-49b8-babb-2d08e6e49221" (UID: "b6d9d7ca-3c71-49b8-babb-2d08e6e49221"). InnerVolumeSpecName "kube-api-access-nck8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:25.067174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.067070    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-xks-ca" pod "b6d9d7ca-3c71-49b8-babb-2d08e6e49221" (UID: "b6d9d7ca-3c71-49b8-babb-2d08e6e49221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:25.166369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.166331    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nck8l\" (UniqueName: \"kubernetes.io/projected/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-kube-api-access-nck8l\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:25.166369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.166370    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9d7ca-3c71-49b8-babb-2d08e6e49221-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:25.809689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.809658    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89" event={"ID":"b6d9d7ca-3c71-49b8-babb-2d08e6e49221","Type":"ContainerDied","Data":"0d9c1b61df78d4e6a48ad1645fc9f80c31326e3ab2125ebaaad00107d90e6d94"}
Nov 02 01:48:25.809851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.809706    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d9c1b61df78d4e6a48ad1645fc9f80c31326e3ab2125ebaaad00107d90e6d94"
Nov 02 01:48:25.809851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:25.809707    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"
Nov 02 01:48:26.827454 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:26.827419    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"]
Nov 02 01:48:26.827981 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:26.827647    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy" containerID="cri-o://3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf" gracePeriod=30
Nov 02 01:48:28.557481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:28.557450    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:29.913718 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:29.913684    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nfdcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:48:29.914131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:29.913751    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:48:32.121204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.121177    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:48:32.255980 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.255944    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-token\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-token\") pod \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") "
Nov 02 01:48:32.256238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256006    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-data\") pod \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") "
Nov 02 01:48:32.256238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256041    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-kube-api-access-c79tx\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-kube-api-access-c79tx\") pod \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") "
Nov 02 01:48:32.256238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256099    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-credential-socket\") pod \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") "
Nov 02 01:48:32.256238 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256153    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-podinfo\") pod \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") "
Nov 02 01:48:32.256484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256280    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-certs\") pod \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") "
Nov 02 01:48:32.256484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256310    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-socket\") pod \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") "
Nov 02 01:48:32.256484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256347    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istiod-ca-cert\") pod \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") "
Nov 02 01:48:32.256484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256390    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-envoy\") pod \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\" (UID: \"80c31c29-3c02-42ed-b6ef-24518ff9fa4a\") "
Nov 02 01:48:32.256484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256482    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-credential-socket" pod "80c31c29-3c02-42ed-b6ef-24518ff9fa4a" (UID: "80c31c29-3c02-42ed-b6ef-24518ff9fa4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.257221 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256641    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-certs" pod "80c31c29-3c02-42ed-b6ef-24518ff9fa4a" (UID: "80c31c29-3c02-42ed-b6ef-24518ff9fa4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.257221 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256831    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-data" pod "80c31c29-3c02-42ed-b6ef-24518ff9fa4a" (UID: "80c31c29-3c02-42ed-b6ef-24518ff9fa4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.257221 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256883    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.257221 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.256903    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.257221 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.257195    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istiod-ca-cert" pod "80c31c29-3c02-42ed-b6ef-24518ff9fa4a" (UID: "80c31c29-3c02-42ed-b6ef-24518ff9fa4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:32.257476 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.257447    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-socket" pod "80c31c29-3c02-42ed-b6ef-24518ff9fa4a" (UID: "80c31c29-3c02-42ed-b6ef-24518ff9fa4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.260955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.259605    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-podinfo" pod "80c31c29-3c02-42ed-b6ef-24518ff9fa4a" (UID: "80c31c29-3c02-42ed-b6ef-24518ff9fa4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:48:32.260955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.259718    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-kube-api-access-c79tx" pod "80c31c29-3c02-42ed-b6ef-24518ff9fa4a" (UID: "80c31c29-3c02-42ed-b6ef-24518ff9fa4a"). InnerVolumeSpecName "kube-api-access-c79tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:32.260955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.259951    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-envoy" pod "80c31c29-3c02-42ed-b6ef-24518ff9fa4a" (UID: "80c31c29-3c02-42ed-b6ef-24518ff9fa4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.267398 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.267378    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-token" pod "80c31c29-3c02-42ed-b6ef-24518ff9fa4a" (UID: "80c31c29-3c02-42ed-b6ef-24518ff9fa4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:32.357747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.357670    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.357747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.357700    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.357747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.357715    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.357747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.357726    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.357747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.357738    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.358020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.357751    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.358020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.357765    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79tx\" (UniqueName: \"kubernetes.io/projected/80c31c29-3c02-42ed-b6ef-24518ff9fa4a-kube-api-access-c79tx\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.865900 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.865866    4912 generic.go:356] "Generic (PLEG): container finished" podID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerID="3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf" exitCode=0
Nov 02 01:48:32.866163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.865933    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"
Nov 02 01:48:32.866163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.865977    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" event={"ID":"80c31c29-3c02-42ed-b6ef-24518ff9fa4a","Type":"ContainerDied","Data":"3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf"}
Nov 02 01:48:32.866163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.866046    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd" event={"ID":"80c31c29-3c02-42ed-b6ef-24518ff9fa4a","Type":"ContainerDied","Data":"da382ac5326c3516f8eccc4954529423fc596eeadcd855c6a017aa26e8e6011c"}
Nov 02 01:48:32.866163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.866086    4912 scope.go:122] "RemoveContainer" containerID="3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf"
Nov 02 01:48:32.878363 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.878342    4912 scope.go:122] "RemoveContainer" containerID="2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e"
Nov 02 01:48:32.888096 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.888043    4912 scope.go:122] "RemoveContainer" containerID="3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf"
Nov 02 01:48:32.888414 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:48:32.888386    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf\": container with ID starting with 3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf not found: ID does not exist" containerID="3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf"
Nov 02 01:48:32.888468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.888427    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf"} err="failed to get container status \"3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf\": rpc error: code = NotFound desc = could not find container \"3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf\": container with ID starting with 3aa26cf7022afdbd0b0b0d11601cb526ee4faae32252ea1e0920c675779ef3bf not found: ID does not exist"
Nov 02 01:48:32.888468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.888452    4912 scope.go:122] "RemoveContainer" containerID="2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e"
Nov 02 01:48:32.888716 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:48:32.888699    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e\": container with ID starting with 2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e not found: ID does not exist" containerID="2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e"
Nov 02 01:48:32.888766 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:32.888722    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e"} err="failed to get container status \"2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e\": rpc error: code = NotFound desc = could not find container \"2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e\": container with ID starting with 2eb0648c9d99e565475f63ae4ca6760e1990e2ac0f2002e604ac9563e3da6a2e not found: ID does not exist"
Nov 02 01:48:33.085610 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:33.085572    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"]
Nov 02 01:48:33.095970 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:33.095939    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nfdcd"]
Nov 02 01:48:33.225753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:33.225669    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" path="/var/lib/kubelet/pods/80c31c29-3c02-42ed-b6ef-24518ff9fa4a/volumes"
Nov 02 01:48:33.358022 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:33.357985    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:35.822894 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.822860    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"]
Nov 02 01:48:35.823623 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.823604    4912 cpu_manager.go:414] "containerMap: removing container" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy"
Nov 02 01:48:35.823722 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.823625    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy"
Nov 02 01:48:35.823722 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.823653    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b6d9d7ca-3c71-49b8-babb-2d08e6e49221" containerName="revoke-keys"
Nov 02 01:48:35.823722 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.823661    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d9d7ca-3c71-49b8-babb-2d08e6e49221" containerName="revoke-keys"
Nov 02 01:48:35.823868 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.823834    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d9d7ca-3c71-49b8-babb-2d08e6e49221" containerName="revoke-keys"
Nov 02 01:48:35.823868 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.823854    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy"
Nov 02 01:48:35.828594 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.828574    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:35.885299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.885265    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"]
Nov 02 01:48:35.896545 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.896510    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b89902-n6zg8\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:35.896733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.896700    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b89902-n6zg8\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:35.896810 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.896752    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp27l\" (UniqueName: \"kubernetes.io/projected/583d6858-964c-49b4-b8fa-6e3a59e16230-kube-api-access-cp27l\") pod \"maas-api-revoke-sub-82b89902-n6zg8\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:35.930016 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.929985    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:35.930214 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.930146    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:48:35.997833 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.997798    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b89902-n6zg8\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:35.997997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.997933    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b89902-n6zg8\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:35.997997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.997993    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp27l\" (UniqueName: \"kubernetes.io/projected/583d6858-964c-49b4-b8fa-6e3a59e16230-kube-api-access-cp27l\") pod \"maas-api-revoke-sub-82b89902-n6zg8\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:35.998531 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:35.998506    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b89902-n6zg8\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:36.000587 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:36.000565    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b89902-n6zg8\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:36.058623 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:36.058593    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp27l\" (UniqueName: \"kubernetes.io/projected/583d6858-964c-49b4-b8fa-6e3a59e16230-kube-api-access-cp27l\") pod \"maas-api-revoke-sub-82b89902-n6zg8\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:36.143199 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:36.143163    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:36.528768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:36.528737    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"]
Nov 02 01:48:36.528958 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:48:36.528928    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583d6858_964c_49b4_b8fa_6e3a59e16230.slice/crio-bf2783af361de8fa7c3126e8f0b3956da2da0e26f8907f9624c52e0cd9366ada WatchSource:0}: Error finding container bf2783af361de8fa7c3126e8f0b3956da2da0e26f8907f9624c52e0cd9366ada: Status 404 returned error can't find the container with id bf2783af361de8fa7c3126e8f0b3956da2da0e26f8907f9624c52e0cd9366ada
Nov 02 01:48:36.904527 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:36.904492    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8" event={"ID":"583d6858-964c-49b4-b8fa-6e3a59e16230","Type":"ContainerStarted","Data":"bf2783af361de8fa7c3126e8f0b3956da2da0e26f8907f9624c52e0cd9366ada"}
Nov 02 01:48:37.913894 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:37.913862    4912 generic.go:356] "Generic (PLEG): container finished" podID="583d6858-964c-49b4-b8fa-6e3a59e16230" containerID="8346520c01dac7bb4fc924334515ff2e63dde8c0ab5f513deac8cca20a5f96d5" exitCode=0
Nov 02 01:48:37.914313 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:37.913951    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8" event={"ID":"583d6858-964c-49b4-b8fa-6e3a59e16230","Type":"ContainerDied","Data":"8346520c01dac7bb4fc924334515ff2e63dde8c0ab5f513deac8cca20a5f96d5"}
Nov 02 01:48:38.191131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:38.189633    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:39.103325 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.103301    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:39.259124 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.259015    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-xks-ca\") pod \"583d6858-964c-49b4-b8fa-6e3a59e16230\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") "
Nov 02 01:48:39.259305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.259140    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583d6858-964c-49b4-b8fa-6e3a59e16230-kube-api-access-cp27l\" (UniqueName: \"kubernetes.io/projected/583d6858-964c-49b4-b8fa-6e3a59e16230-kube-api-access-cp27l\") pod \"583d6858-964c-49b4-b8fa-6e3a59e16230\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") "
Nov 02 01:48:39.259305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.259230    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-openshift-ca\") pod \"583d6858-964c-49b4-b8fa-6e3a59e16230\" (UID: \"583d6858-964c-49b4-b8fa-6e3a59e16230\") "
Nov 02 01:48:39.259984 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.259958    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-openshift-ca" pod "583d6858-964c-49b4-b8fa-6e3a59e16230" (UID: "583d6858-964c-49b4-b8fa-6e3a59e16230"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:39.262986 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.262962    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-xks-ca" pod "583d6858-964c-49b4-b8fa-6e3a59e16230" (UID: "583d6858-964c-49b4-b8fa-6e3a59e16230"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:39.266676 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.266646    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583d6858-964c-49b4-b8fa-6e3a59e16230-kube-api-access-cp27l" pod "583d6858-964c-49b4-b8fa-6e3a59e16230" (UID: "583d6858-964c-49b4-b8fa-6e3a59e16230"). InnerVolumeSpecName "kube-api-access-cp27l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:39.360999 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.360961    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp27l\" (UniqueName: \"kubernetes.io/projected/583d6858-964c-49b4-b8fa-6e3a59e16230-kube-api-access-cp27l\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:39.360999 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.360995    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:39.360999 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.361010    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d6858-964c-49b4-b8fa-6e3a59e16230-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:39.929669 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.929640    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"
Nov 02 01:48:39.929844 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.929683    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8" event={"ID":"583d6858-964c-49b4-b8fa-6e3a59e16230","Type":"ContainerDied","Data":"bf2783af361de8fa7c3126e8f0b3956da2da0e26f8907f9624c52e0cd9366ada"}
Nov 02 01:48:39.929844 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:39.929735    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf2783af361de8fa7c3126e8f0b3956da2da0e26f8907f9624c52e0cd9366ada"
Nov 02 01:48:42.139903 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:48:42.139815    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:48:42.849879 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.849840    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"]
Nov 02 01:48:42.850730 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.850707    4912 cpu_manager.go:414] "containerMap: removing container" podUID="583d6858-964c-49b4-b8fa-6e3a59e16230" containerName="revoke-keys"
Nov 02 01:48:42.850778 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.850735    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583d6858-964c-49b4-b8fa-6e3a59e16230" containerName="revoke-keys"
Nov 02 01:48:42.850826 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.850813    4912 cpu_manager.go:414] "containerMap: removing container" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy"
Nov 02 01:48:42.850858 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.850828    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy"
Nov 02 01:48:42.850983 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.850971    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c31c29-3c02-42ed-b6ef-24518ff9fa4a" containerName="istio-proxy"
Nov 02 01:48:42.851020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.850993    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="583d6858-964c-49b4-b8fa-6e3a59e16230" containerName="revoke-keys"
Nov 02 01:48:42.855136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.855117    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:42.901289 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.901238    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce5b7976-9ddkb\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:42.901441 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.901362    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce5b7976-9ddkb\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:42.901480 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.901442    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6rk\" (UniqueName: \"kubernetes.io/projected/7716e24f-7066-448f-b110-a296f1faf33d-kube-api-access-nt6rk\") pod \"maas-api-revoke-sub-ce5b7976-9ddkb\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:42.911876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.911844    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"]
Nov 02 01:48:42.954770 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.954724    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:42.954970 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:42.954903    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:48:43.002613 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.002578    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce5b7976-9ddkb\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:43.003400 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.003370    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce5b7976-9ddkb\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:43.003571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.003548    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6rk\" (UniqueName: \"kubernetes.io/projected/7716e24f-7066-448f-b110-a296f1faf33d-kube-api-access-nt6rk\") pod \"maas-api-revoke-sub-ce5b7976-9ddkb\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:43.003641 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.003582    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce5b7976-9ddkb\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:43.008288 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.008261    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce5b7976-9ddkb\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:43.065904 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.065866    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6rk\" (UniqueName: \"kubernetes.io/projected/7716e24f-7066-448f-b110-a296f1faf33d-kube-api-access-nt6rk\") pod \"maas-api-revoke-sub-ce5b7976-9ddkb\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:43.170477 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.170446    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:43.268581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.268541    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:43.448021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.447937    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:43.552917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.552885    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"]
Nov 02 01:48:43.553166 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:48:43.553142    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7716e24f_7066_448f_b110_a296f1faf33d.slice/crio-0ddd4f1b64e87139e9a4ecd1ce15af63f9ff723f9a470ae719ae79950f788c21 WatchSource:0}: Error finding container 0ddd4f1b64e87139e9a4ecd1ce15af63f9ff723f9a470ae719ae79950f788c21: Status 404 returned error can't find the container with id 0ddd4f1b64e87139e9a4ecd1ce15af63f9ff723f9a470ae719ae79950f788c21
Nov 02 01:48:43.962929 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.962894    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb" event={"ID":"7716e24f-7066-448f-b110-a296f1faf33d","Type":"ContainerStarted","Data":"4b80cc2e20415e90ab7568a8d05c0b10641ff11ffb580dbbfc2cbf54ae130e5c"}
Nov 02 01:48:43.963133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:43.962950    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb" event={"ID":"7716e24f-7066-448f-b110-a296f1faf33d","Type":"ContainerStarted","Data":"0ddd4f1b64e87139e9a4ecd1ce15af63f9ff723f9a470ae719ae79950f788c21"}
Nov 02 01:48:44.087131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:44.087018    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb" podStartSLOduration=2.087004472 podStartE2EDuration="2.087004472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:44.08562337 +0000 UTC m=+1016.991088728" watchObservedRunningTime="2026-11-02 01:48:44.087004472 +0000 UTC m=+1016.992469828"
Nov 02 01:48:44.972881 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:44.972839    4912 generic.go:356] "Generic (PLEG): container finished" podID="7716e24f-7066-448f-b110-a296f1faf33d" containerID="4b80cc2e20415e90ab7568a8d05c0b10641ff11ffb580dbbfc2cbf54ae130e5c" exitCode=0
Nov 02 01:48:44.973419 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:44.972896    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb" event={"ID":"7716e24f-7066-448f-b110-a296f1faf33d","Type":"ContainerDied","Data":"4b80cc2e20415e90ab7568a8d05c0b10641ff11ffb580dbbfc2cbf54ae130e5c"}
Nov 02 01:48:46.131817 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.131788    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:46.331337 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.331249    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-openshift-ca\") pod \"7716e24f-7066-448f-b110-a296f1faf33d\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") "
Nov 02 01:48:46.331337 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.331307    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7716e24f-7066-448f-b110-a296f1faf33d-kube-api-access-nt6rk\" (UniqueName: \"kubernetes.io/projected/7716e24f-7066-448f-b110-a296f1faf33d-kube-api-access-nt6rk\") pod \"7716e24f-7066-448f-b110-a296f1faf33d\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") "
Nov 02 01:48:46.331583 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.331382    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-xks-ca\") pod \"7716e24f-7066-448f-b110-a296f1faf33d\" (UID: \"7716e24f-7066-448f-b110-a296f1faf33d\") "
Nov 02 01:48:46.331697 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.331674    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-openshift-ca" pod "7716e24f-7066-448f-b110-a296f1faf33d" (UID: "7716e24f-7066-448f-b110-a296f1faf33d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:46.333598 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.333575    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7716e24f-7066-448f-b110-a296f1faf33d-kube-api-access-nt6rk" pod "7716e24f-7066-448f-b110-a296f1faf33d" (UID: "7716e24f-7066-448f-b110-a296f1faf33d"). InnerVolumeSpecName "kube-api-access-nt6rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:46.333689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.333618    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-xks-ca" pod "7716e24f-7066-448f-b110-a296f1faf33d" (UID: "7716e24f-7066-448f-b110-a296f1faf33d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:46.432560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.432523    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:46.432560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.432554    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt6rk\" (UniqueName: \"kubernetes.io/projected/7716e24f-7066-448f-b110-a296f1faf33d-kube-api-access-nt6rk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:46.432763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.432567    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7716e24f-7066-448f-b110-a296f1faf33d-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:46.986808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.986781    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"
Nov 02 01:48:46.986808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.986809    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb" event={"ID":"7716e24f-7066-448f-b110-a296f1faf33d","Type":"ContainerDied","Data":"0ddd4f1b64e87139e9a4ecd1ce15af63f9ff723f9a470ae719ae79950f788c21"}
Nov 02 01:48:46.987029 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:46.986853    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ddd4f1b64e87139e9a4ecd1ce15af63f9ff723f9a470ae719ae79950f788c21"
Nov 02 01:48:48.151508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:48.151451    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:52.945187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:52.945152    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:56.749241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.749202    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"]
Nov 02 01:48:56.750132 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.750113    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7716e24f-7066-448f-b110-a296f1faf33d" containerName="revoke-keys"
Nov 02 01:48:56.750190 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.750136    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7716e24f-7066-448f-b110-a296f1faf33d" containerName="revoke-keys"
Nov 02 01:48:56.750301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.750288    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="7716e24f-7066-448f-b110-a296f1faf33d" containerName="revoke-keys"
Nov 02 01:48:56.755651 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.755630    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:56.813631 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.813603    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"]
Nov 02 01:48:56.850126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.850051    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a9ff421-r4hkq\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:56.850328 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.850304    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gc7w\" (UniqueName: \"kubernetes.io/projected/a8679b55-4cc4-4740-8d0e-c71bbb616175-kube-api-access-4gc7w\") pod \"maas-api-revoke-sub-9a9ff421-r4hkq\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:56.850442 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.850410    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a9ff421-r4hkq\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:56.855778 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.855747    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:56.855970 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.855805    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:48:56.952328 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.952274    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gc7w\" (UniqueName: \"kubernetes.io/projected/a8679b55-4cc4-4740-8d0e-c71bbb616175-kube-api-access-4gc7w\") pod \"maas-api-revoke-sub-9a9ff421-r4hkq\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:56.952990 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.952963    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a9ff421-r4hkq\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:56.953530 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.953508    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a9ff421-r4hkq\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:56.954660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.954417    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a9ff421-r4hkq\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:56.965210 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:56.963558    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a9ff421-r4hkq\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:57.016275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:57.016206    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gc7w\" (UniqueName: \"kubernetes.io/projected/a8679b55-4cc4-4740-8d0e-c71bbb616175-kube-api-access-4gc7w\") pod \"maas-api-revoke-sub-9a9ff421-r4hkq\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:57.072143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:57.072109    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:48:57.467353 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:57.467324    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"]
Nov 02 01:48:57.468178 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:48:57.468147    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8679b55_4cc4_4740_8d0e_c71bbb616175.slice/crio-06356c2217a50ce099da931d21ecf2778a0d41afc52e984c2e6c3affa508b9a4 WatchSource:0}: Error finding container 06356c2217a50ce099da931d21ecf2778a0d41afc52e984c2e6c3affa508b9a4: Status 404 returned error can't find the container with id 06356c2217a50ce099da931d21ecf2778a0d41afc52e984c2e6c3affa508b9a4
Nov 02 01:48:57.847747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:57.847693    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:48:58.069520 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:58.069434    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq" event={"ID":"a8679b55-4cc4-4740-8d0e-c71bbb616175","Type":"ContainerStarted","Data":"a40e52e44340d87d6002b9cdec22f98a9e921457a7dc73652ae54d5cc126fcac"}
Nov 02 01:48:58.069520 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:58.069497    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq" event={"ID":"a8679b55-4cc4-4740-8d0e-c71bbb616175","Type":"ContainerStarted","Data":"06356c2217a50ce099da931d21ecf2778a0d41afc52e984c2e6c3affa508b9a4"}
Nov 02 01:48:58.196126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:58.196050    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq" podStartSLOduration=2.196032093 podStartE2EDuration="2.196032093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:58.186927211 +0000 UTC m=+1031.092392573" watchObservedRunningTime="2026-11-02 01:48:58.196032093 +0000 UTC m=+1031.101497451"
Nov 02 01:48:59.079182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.079145    4912 generic.go:356] "Generic (PLEG): container finished" podID="a8679b55-4cc4-4740-8d0e-c71bbb616175" containerID="a40e52e44340d87d6002b9cdec22f98a9e921457a7dc73652ae54d5cc126fcac" exitCode=0
Nov 02 01:48:59.079657 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.079217    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq" event={"ID":"a8679b55-4cc4-4740-8d0e-c71bbb616175","Type":"ContainerDied","Data":"a40e52e44340d87d6002b9cdec22f98a9e921457a7dc73652ae54d5cc126fcac"}
Nov 02 01:48:59.248209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.248175    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"]
Nov 02 01:48:59.260343 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.260318    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.309639 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.309607    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"]
Nov 02 01:48:59.407951 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.407914    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/8788c8f3-2c96-4503-8897-8fb7ea516c9e-kube-api-access-ldxwz\") pod \"maas-api-revoke-sub-564a41a1-8wsh5\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.408165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.408022    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564a41a1-8wsh5\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.408165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.408089    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564a41a1-8wsh5\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.509008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.508974    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/8788c8f3-2c96-4503-8897-8fb7ea516c9e-kube-api-access-ldxwz\") pod \"maas-api-revoke-sub-564a41a1-8wsh5\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.509208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.509038    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564a41a1-8wsh5\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.509208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.509097    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564a41a1-8wsh5\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.509824 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.509756    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564a41a1-8wsh5\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.511510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.511485    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564a41a1-8wsh5\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.570386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.570355    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/8788c8f3-2c96-4503-8897-8fb7ea516c9e-kube-api-access-ldxwz\") pod \"maas-api-revoke-sub-564a41a1-8wsh5\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.576012 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.575986    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:48:59.956527 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:48:59.956501    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"]
Nov 02 01:48:59.957956 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:48:59.957921    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8788c8f3_2c96_4503_8897_8fb7ea516c9e.slice/crio-a2443242b2e0d29bc901906dcd8047de848254ca3145c6207b98d9c66818a31f WatchSource:0}: Error finding container a2443242b2e0d29bc901906dcd8047de848254ca3145c6207b98d9c66818a31f: Status 404 returned error can't find the container with id a2443242b2e0d29bc901906dcd8047de848254ca3145c6207b98d9c66818a31f
Nov 02 01:49:00.087917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.087881    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5" event={"ID":"8788c8f3-2c96-4503-8897-8fb7ea516c9e","Type":"ContainerStarted","Data":"a2443242b2e0d29bc901906dcd8047de848254ca3145c6207b98d9c66818a31f"}
Nov 02 01:49:00.243676 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.243650    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:49:00.319321 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.319288    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-xks-ca\") pod \"a8679b55-4cc4-4740-8d0e-c71bbb616175\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") "
Nov 02 01:49:00.319488 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.319353    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8679b55-4cc4-4740-8d0e-c71bbb616175-kube-api-access-4gc7w\" (UniqueName: \"kubernetes.io/projected/a8679b55-4cc4-4740-8d0e-c71bbb616175-kube-api-access-4gc7w\") pod \"a8679b55-4cc4-4740-8d0e-c71bbb616175\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") "
Nov 02 01:49:00.319488 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.319386    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-openshift-ca\") pod \"a8679b55-4cc4-4740-8d0e-c71bbb616175\" (UID: \"a8679b55-4cc4-4740-8d0e-c71bbb616175\") "
Nov 02 01:49:00.319770 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.319746    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-openshift-ca" pod "a8679b55-4cc4-4740-8d0e-c71bbb616175" (UID: "a8679b55-4cc4-4740-8d0e-c71bbb616175"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:00.321509 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.321485    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-xks-ca" pod "a8679b55-4cc4-4740-8d0e-c71bbb616175" (UID: "a8679b55-4cc4-4740-8d0e-c71bbb616175"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:00.321604 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.321589    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8679b55-4cc4-4740-8d0e-c71bbb616175-kube-api-access-4gc7w" pod "a8679b55-4cc4-4740-8d0e-c71bbb616175" (UID: "a8679b55-4cc4-4740-8d0e-c71bbb616175"). InnerVolumeSpecName "kube-api-access-4gc7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:00.421267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.421240    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:00.421267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.421264    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gc7w\" (UniqueName: \"kubernetes.io/projected/a8679b55-4cc4-4740-8d0e-c71bbb616175-kube-api-access-4gc7w\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:00.421267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:00.421275    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8679b55-4cc4-4740-8d0e-c71bbb616175-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:01.096814 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:01.096776    4912 generic.go:356] "Generic (PLEG): container finished" podID="8788c8f3-2c96-4503-8897-8fb7ea516c9e" containerID="ca03bc1a816d218354aefe8ae4155af07234cd50bfc48096c4641b249165058a" exitCode=0
Nov 02 01:49:01.097263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:01.096860    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5" event={"ID":"8788c8f3-2c96-4503-8897-8fb7ea516c9e","Type":"ContainerDied","Data":"ca03bc1a816d218354aefe8ae4155af07234cd50bfc48096c4641b249165058a"}
Nov 02 01:49:01.103082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:01.100596    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"
Nov 02 01:49:01.103082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:01.100601    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq" event={"ID":"a8679b55-4cc4-4740-8d0e-c71bbb616175","Type":"ContainerDied","Data":"06356c2217a50ce099da931d21ecf2778a0d41afc52e984c2e6c3affa508b9a4"}
Nov 02 01:49:01.103082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:01.100718    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06356c2217a50ce099da931d21ecf2778a0d41afc52e984c2e6c3affa508b9a4"
Nov 02 01:49:02.261537 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.261513    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:49:02.342848 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.342817    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-openshift-ca\") pod \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") "
Nov 02 01:49:02.342991 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.342904    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-xks-ca\") pod \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") "
Nov 02 01:49:02.342991 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.342968    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8788c8f3-2c96-4503-8897-8fb7ea516c9e-kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/8788c8f3-2c96-4503-8897-8fb7ea516c9e-kube-api-access-ldxwz\") pod \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\" (UID: \"8788c8f3-2c96-4503-8897-8fb7ea516c9e\") "
Nov 02 01:49:02.343257 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.343224    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-openshift-ca" pod "8788c8f3-2c96-4503-8897-8fb7ea516c9e" (UID: "8788c8f3-2c96-4503-8897-8fb7ea516c9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:02.343403 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.343384    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:02.345241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.345218    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8788c8f3-2c96-4503-8897-8fb7ea516c9e-kube-api-access-ldxwz" pod "8788c8f3-2c96-4503-8897-8fb7ea516c9e" (UID: "8788c8f3-2c96-4503-8897-8fb7ea516c9e"). InnerVolumeSpecName "kube-api-access-ldxwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:02.345241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.345233    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-xks-ca" pod "8788c8f3-2c96-4503-8897-8fb7ea516c9e" (UID: "8788c8f3-2c96-4503-8897-8fb7ea516c9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:02.444616 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.444585    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/8788c8f3-2c96-4503-8897-8fb7ea516c9e-kube-api-access-ldxwz\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:02.444616 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.444613    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8788c8f3-2c96-4503-8897-8fb7ea516c9e-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:02.555204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.555167    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"]
Nov 02 01:49:02.555720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.555706    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8788c8f3-2c96-4503-8897-8fb7ea516c9e" containerName="revoke-keys"
Nov 02 01:49:02.555769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.555721    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8788c8f3-2c96-4503-8897-8fb7ea516c9e" containerName="revoke-keys"
Nov 02 01:49:02.555769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.555742    4912 cpu_manager.go:414] "containerMap: removing container" podUID="a8679b55-4cc4-4740-8d0e-c71bbb616175" containerName="revoke-keys"
Nov 02 01:49:02.555769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.555748    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8679b55-4cc4-4740-8d0e-c71bbb616175" containerName="revoke-keys"
Nov 02 01:49:02.555861 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.555844    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8679b55-4cc4-4740-8d0e-c71bbb616175" containerName="revoke-keys"
Nov 02 01:49:02.555861 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.555855    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8788c8f3-2c96-4503-8897-8fb7ea516c9e" containerName="revoke-keys"
Nov 02 01:49:02.559320 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.559296    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.618138 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.618103    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"]
Nov 02 01:49:02.647299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.647263    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8899d465-hxvpj\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.647443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.647335    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4s99\" (UniqueName: \"kubernetes.io/projected/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-kube-api-access-n4s99\") pod \"maas-api-revoke-sub-8899d465-hxvpj\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.647443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.647415    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8899d465-hxvpj\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.666790 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.666725    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:02.749015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.748919    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8899d465-hxvpj\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.749015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.748992    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4s99\" (UniqueName: \"kubernetes.io/projected/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-kube-api-access-n4s99\") pod \"maas-api-revoke-sub-8899d465-hxvpj\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.749269 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.749049    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8899d465-hxvpj\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.749858 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.749832    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8899d465-hxvpj\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.751515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.751490    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8899d465-hxvpj\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.810256 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.810227    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4s99\" (UniqueName: \"kubernetes.io/projected/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-kube-api-access-n4s99\") pod \"maas-api-revoke-sub-8899d465-hxvpj\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:02.875309 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:02.875274    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:03.115311 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:03.115279    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5" event={"ID":"8788c8f3-2c96-4503-8897-8fb7ea516c9e","Type":"ContainerDied","Data":"a2443242b2e0d29bc901906dcd8047de848254ca3145c6207b98d9c66818a31f"}
Nov 02 01:49:03.115482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:03.115320    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"
Nov 02 01:49:03.115482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:03.115326    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2443242b2e0d29bc901906dcd8047de848254ca3145c6207b98d9c66818a31f"
Nov 02 01:49:03.252878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:03.252850    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"]
Nov 02 01:49:03.253781 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:49:03.253757    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49fb19e_9e56_4568_8e6f_0b3c0f3fa1cc.slice/crio-1876f9c40655b71dcf657aad652b21ca514ce801a380af4381b2228b4737bde6 WatchSource:0}: Error finding container 1876f9c40655b71dcf657aad652b21ca514ce801a380af4381b2228b4737bde6: Status 404 returned error can't find the container with id 1876f9c40655b71dcf657aad652b21ca514ce801a380af4381b2228b4737bde6
Nov 02 01:49:04.127013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:04.126966    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj" event={"ID":"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc","Type":"ContainerStarted","Data":"03ca3c0d777bc7c40a3dd0a6e99d6b9353f068dfad2bdc21b3a9dc451bdade6c"}
Nov 02 01:49:04.127490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:04.127035    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj" event={"ID":"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc","Type":"ContainerStarted","Data":"1876f9c40655b71dcf657aad652b21ca514ce801a380af4381b2228b4737bde6"}
Nov 02 01:49:05.137667 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.137622    4912 generic.go:356] "Generic (PLEG): container finished" podID="c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc" containerID="03ca3c0d777bc7c40a3dd0a6e99d6b9353f068dfad2bdc21b3a9dc451bdade6c" exitCode=0
Nov 02 01:49:05.138161 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.137692    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj" event={"ID":"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc","Type":"ContainerDied","Data":"03ca3c0d777bc7c40a3dd0a6e99d6b9353f068dfad2bdc21b3a9dc451bdade6c"}
Nov 02 01:49:05.306703 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.306679    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:05.478934 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.478852    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-xks-ca\") pod \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") "
Nov 02 01:49:05.479119 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.478934    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-openshift-ca\") pod \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") "
Nov 02 01:49:05.479119 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.478958    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-kube-api-access-n4s99\" (UniqueName: \"kubernetes.io/projected/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-kube-api-access-n4s99\") pod \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\" (UID: \"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc\") "
Nov 02 01:49:05.479331 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.479299    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-openshift-ca" pod "c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc" (UID: "c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:05.479508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.479493    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:05.481116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.481085    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-xks-ca" pod "c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc" (UID: "c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:05.481219 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.481199    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-kube-api-access-n4s99" pod "c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc" (UID: "c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc"). InnerVolumeSpecName "kube-api-access-n4s99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:05.580883 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.580854    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:05.580883 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:05.580881    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4s99\" (UniqueName: \"kubernetes.io/projected/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc-kube-api-access-n4s99\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:06.145816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:06.145772    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj" event={"ID":"c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc","Type":"ContainerDied","Data":"1876f9c40655b71dcf657aad652b21ca514ce801a380af4381b2228b4737bde6"}
Nov 02 01:49:06.146293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:06.145832    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1876f9c40655b71dcf657aad652b21ca514ce801a380af4381b2228b4737bde6"
Nov 02 01:49:06.146293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:06.145787    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"
Nov 02 01:49:07.418843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:07.418807    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:11.115781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.115746    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"]
Nov 02 01:49:11.116524 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.116502    4912 cpu_manager.go:414] "containerMap: removing container" podUID="c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc" containerName="revoke-keys"
Nov 02 01:49:11.116617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.116526    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc" containerName="revoke-keys"
Nov 02 01:49:11.116706 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.116693    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc" containerName="revoke-keys"
Nov 02 01:49:11.120340 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.120317    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.148931 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.148898    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-home\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.149112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.148941    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-model-cache\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.149112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.149031    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a49295-57be-4bb4-9148-5bde151f6ecb-tls-certs\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.149230 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.149171    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-tmp-dir\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.149230 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.149226    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzqhr\" (UniqueName: \"kubernetes.io/projected/f6a49295-57be-4bb4-9148-5bde151f6ecb-kube-api-access-mzqhr\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.149331 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.149254    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-dshm\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.203742 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.203704    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"]
Nov 02 01:49:11.223473 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.223442    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dbe849\"/\"distinct-w1-dbe849-kserve-self-signed-certs\""
Nov 02 01:49:11.245932 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.245896    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:11.250165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.250137    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzqhr\" (UniqueName: \"kubernetes.io/projected/f6a49295-57be-4bb4-9148-5bde151f6ecb-kube-api-access-mzqhr\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.250338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.250288    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-dshm\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.250446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.250430    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-home\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.250513 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.250454    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-model-cache\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.250513 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.250500    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a49295-57be-4bb4-9148-5bde151f6ecb-tls-certs\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.250612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.250585    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-tmp-dir\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.250851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.250825    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-model-cache\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.250939 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.250870    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-home\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.250939 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.250888    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-tmp-dir\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.252755 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.252726    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-dshm\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.253088 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.253044    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a49295-57be-4bb4-9148-5bde151f6ecb-tls-certs\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.306262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.306216    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"]
Nov 02 01:49:11.310667 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.310639    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.324820 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.324791    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzqhr\" (UniqueName: \"kubernetes.io/projected/f6a49295-57be-4bb4-9148-5bde151f6ecb-kube-api-access-mzqhr\") pod \"distinct-w1-dbe849-kserve-56d4889976-tkhfj\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.351950 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.351919    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be5a7569-zzqdq\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.352181 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.351962    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be5a7569-zzqdq\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.352181 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.352153    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2ff\" (UniqueName: \"kubernetes.io/projected/8c3a4a0e-4569-4a21-9d23-d60bd4053668-kube-api-access-bj2ff\") pod \"maas-api-revoke-sub-be5a7569-zzqdq\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.364275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.364244    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:11.364406 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.364347    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:49:11.368860 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.368794    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"]
Nov 02 01:49:11.433802 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.433769    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:11.452951 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.452919    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be5a7569-zzqdq\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.453131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.452986    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be5a7569-zzqdq\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.453209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.453131    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2ff\" (UniqueName: \"kubernetes.io/projected/8c3a4a0e-4569-4a21-9d23-d60bd4053668-kube-api-access-bj2ff\") pod \"maas-api-revoke-sub-be5a7569-zzqdq\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.453795 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.453772    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be5a7569-zzqdq\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.462464 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.462435    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be5a7569-zzqdq\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.586547 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.586479    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2ff\" (UniqueName: \"kubernetes.io/projected/8c3a4a0e-4569-4a21-9d23-d60bd4053668-kube-api-access-bj2ff\") pod \"maas-api-revoke-sub-be5a7569-zzqdq\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.630131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.629249    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:11.869085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:11.868103    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"]
Nov 02 01:49:12.068882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.068852    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"]
Nov 02 01:49:12.071111 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:49:12.071080    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3a4a0e_4569_4a21_9d23_d60bd4053668.slice/crio-a13f14f62683629a05fe836c14344d2ea050085e7084190586ab8ca3fc7cb47c WatchSource:0}: Error finding container a13f14f62683629a05fe836c14344d2ea050085e7084190586ab8ca3fc7cb47c: Status 404 returned error can't find the container with id a13f14f62683629a05fe836c14344d2ea050085e7084190586ab8ca3fc7cb47c
Nov 02 01:49:12.138114 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:49:12.138050    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:49:12.193647 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.193553    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" event={"ID":"f6a49295-57be-4bb4-9148-5bde151f6ecb","Type":"ContainerStarted","Data":"efccaa57cc844c3f85de106c91efa468a1aae678e088b1301705462ba7458669"}
Nov 02 01:49:12.193647 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.193603    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" event={"ID":"f6a49295-57be-4bb4-9148-5bde151f6ecb","Type":"ContainerStarted","Data":"1621db7aed7db5519caacb92474eb523a39fe17c97e257e4d5659df0568d2db5"}
Nov 02 01:49:12.195311 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.195279    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq" event={"ID":"8c3a4a0e-4569-4a21-9d23-d60bd4053668","Type":"ContainerStarted","Data":"a13f14f62683629a05fe836c14344d2ea050085e7084190586ab8ca3fc7cb47c"}
Nov 02 01:49:12.241828 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.241789    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:12.313402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.313336    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podStartSLOduration=1.3133169709999999 podStartE2EDuration="1.313316971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:12.310535819 +0000 UTC m=+1045.216001177" watchObservedRunningTime="2026-11-02 01:49:12.313316971 +0000 UTC m=+1045.218782329"
Nov 02 01:49:12.434773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.434736    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:12.662026 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.661987    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"]
Nov 02 01:49:12.666411 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.666388    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.723293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.723213    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"]
Nov 02 01:49:12.748703 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.748675    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bczq4\" (UniqueName: \"kubernetes.io/projected/d1e5e5e0-833c-41a9-8113-87321325edd2-kube-api-access-bczq4\") pod \"maas-api-revoke-sub-719efb0a-jrddr\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.748859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.748724    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719efb0a-jrddr\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.748859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.748816    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719efb0a-jrddr\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.850152 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.850120    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczq4\" (UniqueName: \"kubernetes.io/projected/d1e5e5e0-833c-41a9-8113-87321325edd2-kube-api-access-bczq4\") pod \"maas-api-revoke-sub-719efb0a-jrddr\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.850372 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.850170    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719efb0a-jrddr\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.850372 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.850213    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719efb0a-jrddr\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.850899 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.850876    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719efb0a-jrddr\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.853456 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.853431    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719efb0a-jrddr\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.911797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.911761    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczq4\" (UniqueName: \"kubernetes.io/projected/d1e5e5e0-833c-41a9-8113-87321325edd2-kube-api-access-bczq4\") pod \"maas-api-revoke-sub-719efb0a-jrddr\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:12.980452 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:12.980355    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:13.204832 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:13.204801    4912 generic.go:356] "Generic (PLEG): container finished" podID="8c3a4a0e-4569-4a21-9d23-d60bd4053668" containerID="8a4a214fca5f119220af4fc6de0334fb92a0a89ff0fc40bd122f5b8f678facf1" exitCode=0
Nov 02 01:49:13.205231 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:13.204842    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq" event={"ID":"8c3a4a0e-4569-4a21-9d23-d60bd4053668","Type":"ContainerDied","Data":"8a4a214fca5f119220af4fc6de0334fb92a0a89ff0fc40bd122f5b8f678facf1"}
Nov 02 01:49:13.362405 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:49:13.362372    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e5e5e0_833c_41a9_8113_87321325edd2.slice/crio-dfb9962514ea9e099693fe27879920aee9b7639092f7f67f8425223089d5ee69 WatchSource:0}: Error finding container dfb9962514ea9e099693fe27879920aee9b7639092f7f67f8425223089d5ee69: Status 404 returned error can't find the container with id dfb9962514ea9e099693fe27879920aee9b7639092f7f67f8425223089d5ee69
Nov 02 01:49:13.374816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:13.374790    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"]
Nov 02 01:49:14.213040 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.213000    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr" event={"ID":"d1e5e5e0-833c-41a9-8113-87321325edd2","Type":"ContainerStarted","Data":"b88c82e372332c5c39d57f15678f5d1640780fef8d229ee44d2d4540295f5f60"}
Nov 02 01:49:14.213480 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.213080    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr" event={"ID":"d1e5e5e0-833c-41a9-8113-87321325edd2","Type":"ContainerStarted","Data":"dfb9962514ea9e099693fe27879920aee9b7639092f7f67f8425223089d5ee69"}
Nov 02 01:49:14.330412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.330363    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr" podStartSLOduration=2.330348018 podStartE2EDuration="2.330348018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:14.327991055 +0000 UTC m=+1047.233456441" watchObservedRunningTime="2026-11-02 01:49:14.330348018 +0000 UTC m=+1047.235813375"
Nov 02 01:49:14.377938 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.377912    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:14.468401 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.468316    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-openshift-ca\") pod \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") "
Nov 02 01:49:14.468585 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.468405    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3a4a0e-4569-4a21-9d23-d60bd4053668-kube-api-access-bj2ff\" (UniqueName: \"kubernetes.io/projected/8c3a4a0e-4569-4a21-9d23-d60bd4053668-kube-api-access-bj2ff\") pod \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") "
Nov 02 01:49:14.468585 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.468544    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-xks-ca\") pod \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\" (UID: \"8c3a4a0e-4569-4a21-9d23-d60bd4053668\") "
Nov 02 01:49:14.468734 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.468703    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-openshift-ca" pod "8c3a4a0e-4569-4a21-9d23-d60bd4053668" (UID: "8c3a4a0e-4569-4a21-9d23-d60bd4053668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:14.469075 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.469042    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:14.470829 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.470798    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3a4a0e-4569-4a21-9d23-d60bd4053668-kube-api-access-bj2ff" pod "8c3a4a0e-4569-4a21-9d23-d60bd4053668" (UID: "8c3a4a0e-4569-4a21-9d23-d60bd4053668"). InnerVolumeSpecName "kube-api-access-bj2ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:14.470911 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.470800    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-xks-ca" pod "8c3a4a0e-4569-4a21-9d23-d60bd4053668" (UID: "8c3a4a0e-4569-4a21-9d23-d60bd4053668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:14.570293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.570260    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj2ff\" (UniqueName: \"kubernetes.io/projected/8c3a4a0e-4569-4a21-9d23-d60bd4053668-kube-api-access-bj2ff\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:14.570293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:14.570290    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3a4a0e-4569-4a21-9d23-d60bd4053668-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.223498 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:15.223459    4912 generic.go:356] "Generic (PLEG): container finished" podID="d1e5e5e0-833c-41a9-8113-87321325edd2" containerID="b88c82e372332c5c39d57f15678f5d1640780fef8d229ee44d2d4540295f5f60" exitCode=0
Nov 02 01:49:15.225254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:15.225230    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"
Nov 02 01:49:15.226210 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:15.226183    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr" event={"ID":"d1e5e5e0-833c-41a9-8113-87321325edd2","Type":"ContainerDied","Data":"b88c82e372332c5c39d57f15678f5d1640780fef8d229ee44d2d4540295f5f60"}
Nov 02 01:49:15.226315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:15.226233    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq" event={"ID":"8c3a4a0e-4569-4a21-9d23-d60bd4053668","Type":"ContainerDied","Data":"a13f14f62683629a05fe836c14344d2ea050085e7084190586ab8ca3fc7cb47c"}
Nov 02 01:49:15.226315 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:15.226258    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a13f14f62683629a05fe836c14344d2ea050085e7084190586ab8ca3fc7cb47c"
Nov 02 01:49:16.418674 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.418644    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:16.600133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.598420    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-xks-ca\") pod \"d1e5e5e0-833c-41a9-8113-87321325edd2\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") "
Nov 02 01:49:16.600133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.598657    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e5e5e0-833c-41a9-8113-87321325edd2-kube-api-access-bczq4\" (UniqueName: \"kubernetes.io/projected/d1e5e5e0-833c-41a9-8113-87321325edd2-kube-api-access-bczq4\") pod \"d1e5e5e0-833c-41a9-8113-87321325edd2\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") "
Nov 02 01:49:16.600133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.598742    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-openshift-ca\") pod \"d1e5e5e0-833c-41a9-8113-87321325edd2\" (UID: \"d1e5e5e0-833c-41a9-8113-87321325edd2\") "
Nov 02 01:49:16.603769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.600876    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-openshift-ca" pod "d1e5e5e0-833c-41a9-8113-87321325edd2" (UID: "d1e5e5e0-833c-41a9-8113-87321325edd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:16.604489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.604448    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-xks-ca" pod "d1e5e5e0-833c-41a9-8113-87321325edd2" (UID: "d1e5e5e0-833c-41a9-8113-87321325edd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:16.608715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.608671    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e5e5e0-833c-41a9-8113-87321325edd2-kube-api-access-bczq4" pod "d1e5e5e0-833c-41a9-8113-87321325edd2" (UID: "d1e5e5e0-833c-41a9-8113-87321325edd2"). InnerVolumeSpecName "kube-api-access-bczq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:16.700797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.700755    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:16.700797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.700800    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e5e5e0-833c-41a9-8113-87321325edd2-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:16.700797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:16.700817    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bczq4\" (UniqueName: \"kubernetes.io/projected/d1e5e5e0-833c-41a9-8113-87321325edd2-kube-api-access-bczq4\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.241915 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:17.241890    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"
Nov 02 01:49:17.242075 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:17.241923    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr" event={"ID":"d1e5e5e0-833c-41a9-8113-87321325edd2","Type":"ContainerDied","Data":"dfb9962514ea9e099693fe27879920aee9b7639092f7f67f8425223089d5ee69"}
Nov 02 01:49:17.242075 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:17.241977    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfb9962514ea9e099693fe27879920aee9b7639092f7f67f8425223089d5ee69"
Nov 02 01:49:17.257811 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:17.257779    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:18.219556 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.219511    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:18.619778 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.619747    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"]
Nov 02 01:49:18.620337 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.620324    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8c3a4a0e-4569-4a21-9d23-d60bd4053668" containerName="revoke-keys"
Nov 02 01:49:18.620393 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.620339    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3a4a0e-4569-4a21-9d23-d60bd4053668" containerName="revoke-keys"
Nov 02 01:49:18.620393 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.620353    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d1e5e5e0-833c-41a9-8113-87321325edd2" containerName="revoke-keys"
Nov 02 01:49:18.620393 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.620359    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e5e5e0-833c-41a9-8113-87321325edd2" containerName="revoke-keys"
Nov 02 01:49:18.620486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.620465    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3a4a0e-4569-4a21-9d23-d60bd4053668" containerName="revoke-keys"
Nov 02 01:49:18.620486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.620473    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e5e5e0-833c-41a9-8113-87321325edd2" containerName="revoke-keys"
Nov 02 01:49:18.624137 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.624119    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.681654 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.681624    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"]
Nov 02 01:49:18.723663 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.723632    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpts9\" (UniqueName: \"kubernetes.io/projected/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-kube-api-access-vpts9\") pod \"maas-api-revoke-sub-3ced7c18-vkpvd\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.723834 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.723737    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ced7c18-vkpvd\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.723834 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.723791    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ced7c18-vkpvd\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.725914 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.725892    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:18.725914 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.725905    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:49:18.824530 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.824500    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpts9\" (UniqueName: \"kubernetes.io/projected/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-kube-api-access-vpts9\") pod \"maas-api-revoke-sub-3ced7c18-vkpvd\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.824713 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.824565    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ced7c18-vkpvd\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.824713 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.824605    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ced7c18-vkpvd\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.825228 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.825209    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ced7c18-vkpvd\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.827281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.827260    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ced7c18-vkpvd\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.885778 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.885697    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpts9\" (UniqueName: \"kubernetes.io/projected/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-kube-api-access-vpts9\") pod \"maas-api-revoke-sub-3ced7c18-vkpvd\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:18.937919 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:18.937892    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:19.321710 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:19.321497    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"]
Nov 02 01:49:19.323316 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:49:19.323244    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690cbd4f_02ac_4d53_9bd8_0eb98a249f81.slice/crio-fe6fdf786cbc3ec3c95c7f3370c231fa18507d5c07008e64587d2447b50ad4be WatchSource:0}: Error finding container fe6fdf786cbc3ec3c95c7f3370c231fa18507d5c07008e64587d2447b50ad4be: Status 404 returned error can't find the container with id fe6fdf786cbc3ec3c95c7f3370c231fa18507d5c07008e64587d2447b50ad4be
Nov 02 01:49:20.268047 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:20.268017    4912 generic.go:356] "Generic (PLEG): container finished" podID="690cbd4f-02ac-4d53-9bd8-0eb98a249f81" containerID="ab163fb0c6552b4d16ba6dbaa5a0e835e9e4cfeceba11289c509441c08572f77" exitCode=0
Nov 02 01:49:20.268261 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:20.268101    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd" event={"ID":"690cbd4f-02ac-4d53-9bd8-0eb98a249f81","Type":"ContainerDied","Data":"ab163fb0c6552b4d16ba6dbaa5a0e835e9e4cfeceba11289c509441c08572f77"}
Nov 02 01:49:20.268261 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:20.268152    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd" event={"ID":"690cbd4f-02ac-4d53-9bd8-0eb98a249f81","Type":"ContainerStarted","Data":"fe6fdf786cbc3ec3c95c7f3370c231fa18507d5c07008e64587d2447b50ad4be"}
Nov 02 01:49:21.434441 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.434405    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:21.439441 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.439417    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:21.447612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.447593    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:21.466117 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.466082    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-openshift-ca\") pod \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") "
Nov 02 01:49:21.466274 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.466206    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-kube-api-access-vpts9\" (UniqueName: \"kubernetes.io/projected/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-kube-api-access-vpts9\") pod \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") "
Nov 02 01:49:21.466337 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.466300    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-xks-ca\") pod \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\" (UID: \"690cbd4f-02ac-4d53-9bd8-0eb98a249f81\") "
Nov 02 01:49:21.466485 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.466459    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-openshift-ca" pod "690cbd4f-02ac-4d53-9bd8-0eb98a249f81" (UID: "690cbd4f-02ac-4d53-9bd8-0eb98a249f81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:21.466812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.466789    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:21.468569 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.468547    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-kube-api-access-vpts9" pod "690cbd4f-02ac-4d53-9bd8-0eb98a249f81" (UID: "690cbd4f-02ac-4d53-9bd8-0eb98a249f81"). InnerVolumeSpecName "kube-api-access-vpts9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:21.468644 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.468572    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-xks-ca" pod "690cbd4f-02ac-4d53-9bd8-0eb98a249f81" (UID: "690cbd4f-02ac-4d53-9bd8-0eb98a249f81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:21.568751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.568675    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpts9\" (UniqueName: \"kubernetes.io/projected/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-kube-api-access-vpts9\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:21.568751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.568704    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690cbd4f-02ac-4d53-9bd8-0eb98a249f81-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:21.995199 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.995163    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"]
Nov 02 01:49:21.995985 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.995968    4912 cpu_manager.go:414] "containerMap: removing container" podUID="690cbd4f-02ac-4d53-9bd8-0eb98a249f81" containerName="revoke-keys"
Nov 02 01:49:21.996040 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.995989    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690cbd4f-02ac-4d53-9bd8-0eb98a249f81" containerName="revoke-keys"
Nov 02 01:49:21.996194 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:21.996183    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="690cbd4f-02ac-4d53-9bd8-0eb98a249f81" containerName="revoke-keys"
Nov 02 01:49:22.000320 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.000300    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.059539 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.059494    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:22.075425 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.075372    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c707249b-ncm86\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.075425 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.075425    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9cd5\" (UniqueName: \"kubernetes.io/projected/de06fbac-bbc3-4549-a7c2-78c10830b2c6-kube-api-access-q9cd5\") pod \"maas-api-revoke-sub-c707249b-ncm86\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.075659 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.075509    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c707249b-ncm86\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.077572 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.077014    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"]
Nov 02 01:49:22.176808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.176776    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c707249b-ncm86\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.177010 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.176893    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c707249b-ncm86\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.177010 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.176923    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9cd5\" (UniqueName: \"kubernetes.io/projected/de06fbac-bbc3-4549-a7c2-78c10830b2c6-kube-api-access-q9cd5\") pod \"maas-api-revoke-sub-c707249b-ncm86\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.177644 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.177618    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c707249b-ncm86\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.179432 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.179409    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c707249b-ncm86\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.238151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.238120    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9cd5\" (UniqueName: \"kubernetes.io/projected/de06fbac-bbc3-4549-a7c2-78c10830b2c6-kube-api-access-q9cd5\") pod \"maas-api-revoke-sub-c707249b-ncm86\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.285419 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.285340    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"
Nov 02 01:49:22.285419 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.285376    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd" event={"ID":"690cbd4f-02ac-4d53-9bd8-0eb98a249f81","Type":"ContainerDied","Data":"fe6fdf786cbc3ec3c95c7f3370c231fa18507d5c07008e64587d2447b50ad4be"}
Nov 02 01:49:22.285579 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.285433    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe6fdf786cbc3ec3c95c7f3370c231fa18507d5c07008e64587d2447b50ad4be"
Nov 02 01:49:22.301182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.301160    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:49:22.314444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.314423    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:22.690976 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:49:22.690940    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde06fbac_bbc3_4549_a7c2_78c10830b2c6.slice/crio-4ed4a95ebc2cb0547756575ed2fa6a2a38387f0633dffd2bdd12d0cd20324748 WatchSource:0}: Error finding container 4ed4a95ebc2cb0547756575ed2fa6a2a38387f0633dffd2bdd12d0cd20324748: Status 404 returned error can't find the container with id 4ed4a95ebc2cb0547756575ed2fa6a2a38387f0633dffd2bdd12d0cd20324748
Nov 02 01:49:22.692532 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:22.692502    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"]
Nov 02 01:49:23.295316 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:23.295223    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86" event={"ID":"de06fbac-bbc3-4549-a7c2-78c10830b2c6","Type":"ContainerStarted","Data":"cd46713b5070073419febaae555bd1d0a540bb1a1db93775b29b83f99884a0d6"}
Nov 02 01:49:23.295316 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:23.295275    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86" event={"ID":"de06fbac-bbc3-4549-a7c2-78c10830b2c6","Type":"ContainerStarted","Data":"4ed4a95ebc2cb0547756575ed2fa6a2a38387f0633dffd2bdd12d0cd20324748"}
Nov 02 01:49:23.411578 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:23.411517    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86" podStartSLOduration=2.411502911 podStartE2EDuration="2.411502911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:23.410399671 +0000 UTC m=+1056.315865028" watchObservedRunningTime="2026-11-02 01:49:23.411502911 +0000 UTC m=+1056.316968267"
Nov 02 01:49:24.304050 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:24.304017    4912 generic.go:356] "Generic (PLEG): container finished" podID="de06fbac-bbc3-4549-a7c2-78c10830b2c6" containerID="cd46713b5070073419febaae555bd1d0a540bb1a1db93775b29b83f99884a0d6" exitCode=0
Nov 02 01:49:24.304560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:24.304100    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86" event={"ID":"de06fbac-bbc3-4549-a7c2-78c10830b2c6","Type":"ContainerDied","Data":"cd46713b5070073419febaae555bd1d0a540bb1a1db93775b29b83f99884a0d6"}
Nov 02 01:49:25.453580 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.453555    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:25.514969 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.514926    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-xks-ca\") pod \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") "
Nov 02 01:49:25.515185 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.515095    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-openshift-ca\") pod \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") "
Nov 02 01:49:25.515185 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.515153    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de06fbac-bbc3-4549-a7c2-78c10830b2c6-kube-api-access-q9cd5\" (UniqueName: \"kubernetes.io/projected/de06fbac-bbc3-4549-a7c2-78c10830b2c6-kube-api-access-q9cd5\") pod \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\" (UID: \"de06fbac-bbc3-4549-a7c2-78c10830b2c6\") "
Nov 02 01:49:25.515502 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.515469    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-openshift-ca" pod "de06fbac-bbc3-4549-a7c2-78c10830b2c6" (UID: "de06fbac-bbc3-4549-a7c2-78c10830b2c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:25.515630 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.515615    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:25.517979 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.517955    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-xks-ca" pod "de06fbac-bbc3-4549-a7c2-78c10830b2c6" (UID: "de06fbac-bbc3-4549-a7c2-78c10830b2c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:25.518129 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.517982    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de06fbac-bbc3-4549-a7c2-78c10830b2c6-kube-api-access-q9cd5" pod "de06fbac-bbc3-4549-a7c2-78c10830b2c6" (UID: "de06fbac-bbc3-4549-a7c2-78c10830b2c6"). InnerVolumeSpecName "kube-api-access-q9cd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:25.617119 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.617088    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de06fbac-bbc3-4549-a7c2-78c10830b2c6-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:25.617274 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:25.617126    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9cd5\" (UniqueName: \"kubernetes.io/projected/de06fbac-bbc3-4549-a7c2-78c10830b2c6-kube-api-access-q9cd5\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.319786 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:26.319755    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86" event={"ID":"de06fbac-bbc3-4549-a7c2-78c10830b2c6","Type":"ContainerDied","Data":"4ed4a95ebc2cb0547756575ed2fa6a2a38387f0633dffd2bdd12d0cd20324748"}
Nov 02 01:49:26.319786 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:26.319792    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"
Nov 02 01:49:26.320017 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:26.319811    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed4a95ebc2cb0547756575ed2fa6a2a38387f0633dffd2bdd12d0cd20324748"
Nov 02 01:49:26.860339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:26.860303    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:30.464400 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.464363    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"]
Nov 02 01:49:30.464930 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.464917    4912 cpu_manager.go:414] "containerMap: removing container" podUID="de06fbac-bbc3-4549-a7c2-78c10830b2c6" containerName="revoke-keys"
Nov 02 01:49:30.464975 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.464931    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de06fbac-bbc3-4549-a7c2-78c10830b2c6" containerName="revoke-keys"
Nov 02 01:49:30.465027 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.465018    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="de06fbac-bbc3-4549-a7c2-78c10830b2c6" containerName="revoke-keys"
Nov 02 01:49:30.468175 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.468154    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.521298 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.521021    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:30.521298 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.521209    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:49:30.568904 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.568867    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abc4c43c-7nv82\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.569100 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.568934    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abc4c43c-7nv82\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.569151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.569108    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmf8\" (UniqueName: \"kubernetes.io/projected/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-kube-api-access-2zmf8\") pod \"maas-api-revoke-sub-abc4c43c-7nv82\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.574545 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.574522    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"]
Nov 02 01:49:30.669813 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.669777    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abc4c43c-7nv82\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.669989 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.669857    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abc4c43c-7nv82\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.670036 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.669983    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmf8\" (UniqueName: \"kubernetes.io/projected/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-kube-api-access-2zmf8\") pod \"maas-api-revoke-sub-abc4c43c-7nv82\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.670548 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.670520    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abc4c43c-7nv82\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.672425 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.672407    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abc4c43c-7nv82\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.732125 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.732030    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmf8\" (UniqueName: \"kubernetes.io/projected/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-kube-api-access-2zmf8\") pod \"maas-api-revoke-sub-abc4c43c-7nv82\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:30.782050 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:30.782020    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:31.159117 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:31.159090    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"]
Nov 02 01:49:31.159800 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:49:31.159772    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06eee3ae_1121_44f4_af47_4b47a2dd0ab3.slice/crio-491d0f154ae566e9d88b5f9338a6c8d27bd0c690297c164b73d1ca7ef4989856 WatchSource:0}: Error finding container 491d0f154ae566e9d88b5f9338a6c8d27bd0c690297c164b73d1ca7ef4989856: Status 404 returned error can't find the container with id 491d0f154ae566e9d88b5f9338a6c8d27bd0c690297c164b73d1ca7ef4989856
Nov 02 01:49:31.357080 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:31.357019    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82" event={"ID":"06eee3ae-1121-44f4-af47-4b47a2dd0ab3","Type":"ContainerStarted","Data":"491d0f154ae566e9d88b5f9338a6c8d27bd0c690297c164b73d1ca7ef4989856"}
Nov 02 01:49:31.650047 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:31.650009    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:32.366736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:32.366704    4912 generic.go:356] "Generic (PLEG): container finished" podID="06eee3ae-1121-44f4-af47-4b47a2dd0ab3" containerID="947029259c3c754ae99cecfd89f9b7f2ae59603c0666c9075ff508c777d702a1" exitCode=0
Nov 02 01:49:32.366931 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:32.366787    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82" event={"ID":"06eee3ae-1121-44f4-af47-4b47a2dd0ab3","Type":"ContainerDied","Data":"947029259c3c754ae99cecfd89f9b7f2ae59603c0666c9075ff508c777d702a1"}
Nov 02 01:49:33.531634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.531611    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:33.606593 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.606559    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-xks-ca\") pod \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") "
Nov 02 01:49:33.606782 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.606637    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-openshift-ca\") pod \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") "
Nov 02 01:49:33.606862 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.606818    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-kube-api-access-2zmf8\" (UniqueName: \"kubernetes.io/projected/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-kube-api-access-2zmf8\") pod \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\" (UID: \"06eee3ae-1121-44f4-af47-4b47a2dd0ab3\") "
Nov 02 01:49:33.607048 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.607023    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-openshift-ca" pod "06eee3ae-1121-44f4-af47-4b47a2dd0ab3" (UID: "06eee3ae-1121-44f4-af47-4b47a2dd0ab3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:33.607249 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.607234    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.608920 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.608898    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-xks-ca" pod "06eee3ae-1121-44f4-af47-4b47a2dd0ab3" (UID: "06eee3ae-1121-44f4-af47-4b47a2dd0ab3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:33.609043 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.608951    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-kube-api-access-2zmf8" pod "06eee3ae-1121-44f4-af47-4b47a2dd0ab3" (UID: "06eee3ae-1121-44f4-af47-4b47a2dd0ab3"). InnerVolumeSpecName "kube-api-access-2zmf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:33.709075 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.708981    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zmf8\" (UniqueName: \"kubernetes.io/projected/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-kube-api-access-2zmf8\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.709075 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:33.709020    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06eee3ae-1121-44f4-af47-4b47a2dd0ab3-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.387683 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:34.387653    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"
Nov 02 01:49:34.387890 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:34.387646    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82" event={"ID":"06eee3ae-1121-44f4-af47-4b47a2dd0ab3","Type":"ContainerDied","Data":"491d0f154ae566e9d88b5f9338a6c8d27bd0c690297c164b73d1ca7ef4989856"}
Nov 02 01:49:34.387890 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:34.387794    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491d0f154ae566e9d88b5f9338a6c8d27bd0c690297c164b73d1ca7ef4989856"
Nov 02 01:49:36.168714 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.168678    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"]
Nov 02 01:49:36.169581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.169561    4912 cpu_manager.go:414] "containerMap: removing container" podUID="06eee3ae-1121-44f4-af47-4b47a2dd0ab3" containerName="revoke-keys"
Nov 02 01:49:36.169632 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.169585    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06eee3ae-1121-44f4-af47-4b47a2dd0ab3" containerName="revoke-keys"
Nov 02 01:49:36.169737 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.169726    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="06eee3ae-1121-44f4-af47-4b47a2dd0ab3" containerName="revoke-keys"
Nov 02 01:49:36.174111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.174092    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.226266 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.226240    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-0ebdb3-openshift-default-dockercfg-hkbq9\""
Nov 02 01:49:36.285289 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.285253    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"]
Nov 02 01:49:36.335503 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.335468    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.335676 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.335510    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.335676 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.335560    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.335676 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.335604    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xt5k\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-kube-api-access-2xt5k\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.335676 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.335672    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe4c650-474e-4284-8c44-0c971f95efe5-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.335920 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.335691    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe4c650-474e-4284-8c44-0c971f95efe5-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.335920 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.335799    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.335920 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.335833    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.335920 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.335863    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.393762 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.393715    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:36.437227 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437143    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437227 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437190    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437229    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437269    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437307    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437336    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437373    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xt5k\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-kube-api-access-2xt5k\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437418    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe4c650-474e-4284-8c44-0c971f95efe5-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437439    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe4c650-474e-4284-8c44-0c971f95efe5-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437718    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437818    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.437942 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437919    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.438000 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.437979    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.438426 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.438400    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe4c650-474e-4284-8c44-0c971f95efe5-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.440185 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.440164    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.440275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.440234    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe4c650-474e-4284-8c44-0c971f95efe5-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.497991 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.497954    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.498240 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.498138    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xt5k\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-kube-api-access-2xt5k\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:36.791422 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:36.791331    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:37.226664 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:37.226636    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"]
Nov 02 01:49:37.259004 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:49:37.258975    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe4c650_474e_4284_8c44_0c971f95efe5.slice/crio-7ec53cc8606115cd1ed3461f112ba2d9d0847f0ae5f2ce3d9271edf4b626fd23 WatchSource:0}: Error finding container 7ec53cc8606115cd1ed3461f112ba2d9d0847f0ae5f2ce3d9271edf4b626fd23: Status 404 returned error can't find the container with id 7ec53cc8606115cd1ed3461f112ba2d9d0847f0ae5f2ce3d9271edf4b626fd23
Nov 02 01:49:37.261649 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:37.261610    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:49:37.261757 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:37.261701    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:49:37.261757 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:37.261744    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:49:37.418114 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:37.418048    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" event={"ID":"abe4c650-474e-4284-8c44-0c971f95efe5","Type":"ContainerStarted","Data":"8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9"}
Nov 02 01:49:37.418297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:37.418145    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" event={"ID":"abe4c650-474e-4284-8c44-0c971f95efe5","Type":"ContainerStarted","Data":"7ec53cc8606115cd1ed3461f112ba2d9d0847f0ae5f2ce3d9271edf4b626fd23"}
Nov 02 01:49:37.540972 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:37.540856    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" podStartSLOduration=1.540842187 podStartE2EDuration="1.540842187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:37.539178901 +0000 UTC m=+1070.444644259" watchObservedRunningTime="2026-11-02 01:49:37.540842187 +0000 UTC m=+1070.446307544"
Nov 02 01:49:37.792235 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:37.792145    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:38.477512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.477462    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"]
Nov 02 01:49:38.495035 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.494457    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.536147 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.536113    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"]
Nov 02 01:49:38.582182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.580417    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:38.582182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.581283    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:49:38.589658 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.589618    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcg9\" (UniqueName: \"kubernetes.io/projected/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-kube-api-access-vwcg9\") pod \"maas-api-revoke-sub-2ec82e0e-g8nvr\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.589842 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.589743    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec82e0e-g8nvr\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.589842 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.589771    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec82e0e-g8nvr\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.690735 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.690693    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcg9\" (UniqueName: \"kubernetes.io/projected/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-kube-api-access-vwcg9\") pod \"maas-api-revoke-sub-2ec82e0e-g8nvr\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.690912 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.690813    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec82e0e-g8nvr\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.690912 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.690840    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec82e0e-g8nvr\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.691973 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.691864    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec82e0e-g8nvr\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.693791 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.693766    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec82e0e-g8nvr\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.755923 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.755845    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcg9\" (UniqueName: \"kubernetes.io/projected/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-kube-api-access-vwcg9\") pod \"maas-api-revoke-sub-2ec82e0e-g8nvr\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:38.793698 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.793644    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:49:38.793896 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.793704    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:49:38.813998 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:38.813957    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:39.291095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:39.288957    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"]
Nov 02 01:49:39.298636 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:49:39.298528    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027f1ebf_b2bc_4f43_a9ce_de3cce41157d.slice/crio-9e8d2384c56d98c59cd3857b7d17757ec38c4edb8e7cd91014c00d066af27b8a WatchSource:0}: Error finding container 9e8d2384c56d98c59cd3857b7d17757ec38c4edb8e7cd91014c00d066af27b8a: Status 404 returned error can't find the container with id 9e8d2384c56d98c59cd3857b7d17757ec38c4edb8e7cd91014c00d066af27b8a
Nov 02 01:49:39.436281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:39.436243    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr" event={"ID":"027f1ebf-b2bc-4f43-a9ce-de3cce41157d","Type":"ContainerStarted","Data":"9e8d2384c56d98c59cd3857b7d17757ec38c4edb8e7cd91014c00d066af27b8a"}
Nov 02 01:49:39.792967 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:39.792934    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:49:39.793350 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:39.792982    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:49:40.446232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:40.446196    4912 generic.go:356] "Generic (PLEG): container finished" podID="027f1ebf-b2bc-4f43-a9ce-de3cce41157d" containerID="c3a0889d289b6429e25c83fb9c0f1507e6486adc52d8282d59327601bb5a3e5b" exitCode=0
Nov 02 01:49:40.446375 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:40.446283    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr" event={"ID":"027f1ebf-b2bc-4f43-a9ce-de3cce41157d","Type":"ContainerDied","Data":"c3a0889d289b6429e25c83fb9c0f1507e6486adc52d8282d59327601bb5a3e5b"}
Nov 02 01:49:40.792130 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:40.792034    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:49:40.792276 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:40.792125    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:49:41.690572 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.690543    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:41.739504 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.739426    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-xks-ca\") pod \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") "
Nov 02 01:49:41.739650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.739536    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-openshift-ca\") pod \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") "
Nov 02 01:49:41.739650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.739582    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-kube-api-access-vwcg9\" (UniqueName: \"kubernetes.io/projected/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-kube-api-access-vwcg9\") pod \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\" (UID: \"027f1ebf-b2bc-4f43-a9ce-de3cce41157d\") "
Nov 02 01:49:41.739857 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.739835    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-openshift-ca" pod "027f1ebf-b2bc-4f43-a9ce-de3cce41157d" (UID: "027f1ebf-b2bc-4f43-a9ce-de3cce41157d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:41.739962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.739938    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:41.741590 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.741567    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-xks-ca" pod "027f1ebf-b2bc-4f43-a9ce-de3cce41157d" (UID: "027f1ebf-b2bc-4f43-a9ce-de3cce41157d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:41.741855 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.741832    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-kube-api-access-vwcg9" pod "027f1ebf-b2bc-4f43-a9ce-de3cce41157d" (UID: "027f1ebf-b2bc-4f43-a9ce-de3cce41157d"). InnerVolumeSpecName "kube-api-access-vwcg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:41.773320 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.766353    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:41.793083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.792334    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:49:41.793083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.792394    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:49:41.841111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.841073    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:41.841111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:41.841110    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwcg9\" (UniqueName: \"kubernetes.io/projected/027f1ebf-b2bc-4f43-a9ce-de3cce41157d-kube-api-access-vwcg9\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:42.137824 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:49:42.137790    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:49:42.482974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:42.482882    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"
Nov 02 01:49:42.482974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:42.482918    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr" event={"ID":"027f1ebf-b2bc-4f43-a9ce-de3cce41157d","Type":"ContainerDied","Data":"9e8d2384c56d98c59cd3857b7d17757ec38c4edb8e7cd91014c00d066af27b8a"}
Nov 02 01:49:42.482974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:42.482967    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e8d2384c56d98c59cd3857b7d17757ec38c4edb8e7cd91014c00d066af27b8a"
Nov 02 01:49:42.792609 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:42.792522    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:49:42.792609 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:42.792589    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:49:43.630776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:43.630743    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:44.104122 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:44.104009    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:44.104597 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:44.104440    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:44.105430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:44.105410    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 01:49:46.317666 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.317629    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"]
Nov 02 01:49:46.318237 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.318223    4912 cpu_manager.go:414] "containerMap: removing container" podUID="027f1ebf-b2bc-4f43-a9ce-de3cce41157d" containerName="revoke-keys"
Nov 02 01:49:46.318286 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.318238    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027f1ebf-b2bc-4f43-a9ce-de3cce41157d" containerName="revoke-keys"
Nov 02 01:49:46.318348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.318339    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="027f1ebf-b2bc-4f43-a9ce-de3cce41157d" containerName="revoke-keys"
Nov 02 01:49:46.321874 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.321856    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.375120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.375028    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:46.412296 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.412262    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptlxp\" (UniqueName: \"kubernetes.io/projected/988cd1dc-c4db-47fd-934d-d19147ecd644-kube-api-access-ptlxp\") pod \"maas-api-revoke-sub-bdfda3fd-zmmn2\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.412484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.412361    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdfda3fd-zmmn2\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.412543 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.412498    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdfda3fd-zmmn2\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.424432 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.424404    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:49:46.428303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.428274    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"]
Nov 02 01:49:46.513849 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.513810    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptlxp\" (UniqueName: \"kubernetes.io/projected/988cd1dc-c4db-47fd-934d-d19147ecd644-kube-api-access-ptlxp\") pod \"maas-api-revoke-sub-bdfda3fd-zmmn2\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.514086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.513899    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdfda3fd-zmmn2\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.514086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.513959    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdfda3fd-zmmn2\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.514720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.514696    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdfda3fd-zmmn2\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.516627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.516599    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdfda3fd-zmmn2\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.575741 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.575657    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptlxp\" (UniqueName: \"kubernetes.io/projected/988cd1dc-c4db-47fd-934d-d19147ecd644-kube-api-access-ptlxp\") pod \"maas-api-revoke-sub-bdfda3fd-zmmn2\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.635071 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.635032    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:46.855415 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:46.855327    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:47.016988 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:49:47.016955    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988cd1dc_c4db_47fd_934d_d19147ecd644.slice/crio-7a9107ab535735cd8f338c1028537188b5579ecc2731cd90a8cae67daed7bba4 WatchSource:0}: Error finding container 7a9107ab535735cd8f338c1028537188b5579ecc2731cd90a8cae67daed7bba4: Status 404 returned error can't find the container with id 7a9107ab535735cd8f338c1028537188b5579ecc2731cd90a8cae67daed7bba4
Nov 02 01:49:47.017591 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:47.017518    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"]
Nov 02 01:49:47.530735 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:47.530698    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2" event={"ID":"988cd1dc-c4db-47fd-934d-d19147ecd644","Type":"ContainerStarted","Data":"bd86258dd245d92185ec4d3532d7b794910a2b6ad37143aa564a132af9fc177f"}
Nov 02 01:49:47.531156 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:47.530754    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2" event={"ID":"988cd1dc-c4db-47fd-934d-d19147ecd644","Type":"ContainerStarted","Data":"7a9107ab535735cd8f338c1028537188b5579ecc2731cd90a8cae67daed7bba4"}
Nov 02 01:49:47.649316 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:47.649266    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2" podStartSLOduration=1.649251815 podStartE2EDuration="1.649251815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:47.645416149 +0000 UTC m=+1080.550881507" watchObservedRunningTime="2026-11-02 01:49:47.649251815 +0000 UTC m=+1080.554717171"
Nov 02 01:49:48.540685 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:48.540651    4912 generic.go:356] "Generic (PLEG): container finished" podID="988cd1dc-c4db-47fd-934d-d19147ecd644" containerID="bd86258dd245d92185ec4d3532d7b794910a2b6ad37143aa564a132af9fc177f" exitCode=0
Nov 02 01:49:48.541134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:48.540731    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2" event={"ID":"988cd1dc-c4db-47fd-934d-d19147ecd644","Type":"ContainerDied","Data":"bd86258dd245d92185ec4d3532d7b794910a2b6ad37143aa564a132af9fc177f"}
Nov 02 01:49:49.715030 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.715004    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:49.849126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.848979    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-xks-ca\") pod \"988cd1dc-c4db-47fd-934d-d19147ecd644\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") "
Nov 02 01:49:49.849305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.849185    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-openshift-ca\") pod \"988cd1dc-c4db-47fd-934d-d19147ecd644\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") "
Nov 02 01:49:49.849412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.849392    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988cd1dc-c4db-47fd-934d-d19147ecd644-kube-api-access-ptlxp\" (UniqueName: \"kubernetes.io/projected/988cd1dc-c4db-47fd-934d-d19147ecd644-kube-api-access-ptlxp\") pod \"988cd1dc-c4db-47fd-934d-d19147ecd644\" (UID: \"988cd1dc-c4db-47fd-934d-d19147ecd644\") "
Nov 02 01:49:49.849788 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.849577    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-openshift-ca" pod "988cd1dc-c4db-47fd-934d-d19147ecd644" (UID: "988cd1dc-c4db-47fd-934d-d19147ecd644"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:49.850215 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.850191    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.852212 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.852176    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988cd1dc-c4db-47fd-934d-d19147ecd644-kube-api-access-ptlxp" pod "988cd1dc-c4db-47fd-934d-d19147ecd644" (UID: "988cd1dc-c4db-47fd-934d-d19147ecd644"). InnerVolumeSpecName "kube-api-access-ptlxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:49.855112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.851776    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-xks-ca" pod "988cd1dc-c4db-47fd-934d-d19147ecd644" (UID: "988cd1dc-c4db-47fd-934d-d19147ecd644"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:49.950927 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.950895    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptlxp\" (UniqueName: \"kubernetes.io/projected/988cd1dc-c4db-47fd-934d-d19147ecd644-kube-api-access-ptlxp\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.950927 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:49.950923    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988cd1dc-c4db-47fd-934d-d19147ecd644-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:50.556192 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:50.556161    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"
Nov 02 01:49:50.556368 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:50.556199    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2" event={"ID":"988cd1dc-c4db-47fd-934d-d19147ecd644","Type":"ContainerDied","Data":"7a9107ab535735cd8f338c1028537188b5579ecc2731cd90a8cae67daed7bba4"}
Nov 02 01:49:50.556368 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:50.556249    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a9107ab535735cd8f338c1028537188b5579ecc2731cd90a8cae67daed7bba4"
Nov 02 01:49:51.655316 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:51.655277    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:49:56.448583 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:49:56.448545    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:01.352770 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.352735    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:01.525560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.525523    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"]
Nov 02 01:50:01.526095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.526081    4912 cpu_manager.go:414] "containerMap: removing container" podUID="988cd1dc-c4db-47fd-934d-d19147ecd644" containerName="revoke-keys"
Nov 02 01:50:01.526156 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.526097    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988cd1dc-c4db-47fd-934d-d19147ecd644" containerName="revoke-keys"
Nov 02 01:50:01.526213 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.526204    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="988cd1dc-c4db-47fd-934d-d19147ecd644" containerName="revoke-keys"
Nov 02 01:50:01.529857 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.529842    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.607923 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.602667    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"]
Nov 02 01:50:01.631892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.631862    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"distinct-w3-061fab-kserve-self-signed-certs\""
Nov 02 01:50:01.646781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.646735    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:01.682203 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.682167    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-home\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.682419 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.682216    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-model-cache\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.682419 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.682316    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-dshm\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.682419 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.682355    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tls-certs\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.682733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.682499    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tmp-dir\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.682733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.682522    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f842t\" (UniqueName: \"kubernetes.io/projected/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-kube-api-access-f842t\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.783758 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.783725    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tmp-dir\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.783962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.783771    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f842t\" (UniqueName: \"kubernetes.io/projected/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-kube-api-access-f842t\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.783962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.783850    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-home\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.783962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.783900    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-model-cache\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.783962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.783942    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-dshm\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.784244 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.783978    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tls-certs\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.784244 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.784184    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tmp-dir\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.784244 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.784226    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-home\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.784410 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.784309    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-model-cache\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.786421 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.786398    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-dshm\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.786605 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.786586    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tls-certs\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:01.844939 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:01.844901    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f842t\" (UniqueName: \"kubernetes.io/projected/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-kube-api-access-f842t\") pod \"distinct-w3-061fab-kserve-df579b748-28b2c\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:02.142922 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:02.142884    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:02.529781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:02.529745    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"]
Nov 02 01:50:02.530203 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:50:02.529996    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36ceb9dd_8c34_4493_b89e_9dfa1a05948d.slice/crio-927ffb98e731adae776691359e713a038f1a1259664a4f528fc8dc7c59cad1d5 WatchSource:0}: Error finding container 927ffb98e731adae776691359e713a038f1a1259664a4f528fc8dc7c59cad1d5: Status 404 returned error can't find the container with id 927ffb98e731adae776691359e713a038f1a1259664a4f528fc8dc7c59cad1d5
Nov 02 01:50:02.651479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:02.651442    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" event={"ID":"36ceb9dd-8c34-4493-b89e-9dfa1a05948d","Type":"ContainerStarted","Data":"5af6cd14af59ac97a39da125cd5e5494252e47e078cca04ac4133002b800b935"}
Nov 02 01:50:02.651655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:02.651493    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" event={"ID":"36ceb9dd-8c34-4493-b89e-9dfa1a05948d","Type":"ContainerStarted","Data":"927ffb98e731adae776691359e713a038f1a1259664a4f528fc8dc7c59cad1d5"}
Nov 02 01:50:02.768479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:02.768421    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podStartSLOduration=1.7684029730000002 podStartE2EDuration="1.768402973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:02.767798574 +0000 UTC m=+1095.673263936" watchObservedRunningTime="2026-11-02 01:50:02.768402973 +0000 UTC m=+1095.673868329"
Nov 02 01:50:03.143417 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:03.143388    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:06.151724 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:06.151685    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:07.655837 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.655793    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"]
Nov 02 01:50:07.660250 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.660224    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.763000 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.762964    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c62671\"/\"default-dockercfg-z5z9w\""
Nov 02 01:50:07.763229 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.763005    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c62671\"/\"kube-root-ca.crt\""
Nov 02 01:50:07.763229 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.763029    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c62671\"/\"distinct-w2-c62671-kserve-self-signed-certs\""
Nov 02 01:50:07.763229 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.763102    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c62671\"/\"openshift-service-ca.crt\""
Nov 02 01:50:07.769380 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.769347    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"]
Nov 02 01:50:07.775325 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.775296    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:07.850838 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.850795    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tls-certs\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.851027 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.850848    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-home\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.851092 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.851036    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tmp-dir\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.851134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.851092    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9l5\" (UniqueName: \"kubernetes.io/projected/eff3e499-a2c2-4e76-a04b-7f054aef3db7-kube-api-access-8l9l5\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.851216 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.851197    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-model-cache\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.851251 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.851229    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-dshm\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.952595 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.952504    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tls-certs\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.952595 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.952541    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-home\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.952856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.952744    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tmp-dir\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.952856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.952790    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9l5\" (UniqueName: \"kubernetes.io/projected/eff3e499-a2c2-4e76-a04b-7f054aef3db7-kube-api-access-8l9l5\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.952960 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.952879    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-model-cache\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.952960 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.952911    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-dshm\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.953084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.952964    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-home\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.953084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.953025    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tmp-dir\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.953402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.953374    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-model-cache\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.955407 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.955377    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-dshm\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:07.955843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:07.955822    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tls-certs\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:08.014449 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:08.014411    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9l5\" (UniqueName: \"kubernetes.io/projected/eff3e499-a2c2-4e76-a04b-7f054aef3db7-kube-api-access-8l9l5\") pod \"distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:08.276797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:08.276710    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:08.663002 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:50:08.662973    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff3e499_a2c2_4e76_a04b_7f054aef3db7.slice/crio-bb90ca5c0a3b7bed7f325d86611464d4d354dda620bb383f65864fd136da8fbd WatchSource:0}: Error finding container bb90ca5c0a3b7bed7f325d86611464d4d354dda620bb383f65864fd136da8fbd: Status 404 returned error can't find the container with id bb90ca5c0a3b7bed7f325d86611464d4d354dda620bb383f65864fd136da8fbd
Nov 02 01:50:08.664843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:08.664813    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"]
Nov 02 01:50:08.703912 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:08.703882    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" event={"ID":"eff3e499-a2c2-4e76-a04b-7f054aef3db7","Type":"ContainerStarted","Data":"bb90ca5c0a3b7bed7f325d86611464d4d354dda620bb383f65864fd136da8fbd"}
Nov 02 01:50:09.724322 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:09.724278    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" event={"ID":"eff3e499-a2c2-4e76-a04b-7f054aef3db7","Type":"ContainerStarted","Data":"86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737"}
Nov 02 01:50:09.843457 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:09.843397    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podStartSLOduration=2.843378655 podStartE2EDuration="2.843378655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:09.840787585 +0000 UTC m=+1102.746252943" watchObservedRunningTime="2026-11-02 01:50:09.843378655 +0000 UTC m=+1102.748844013"
Nov 02 01:50:10.276799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:10.276762    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:10.972861 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:10.972811    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:11.740822 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.740790    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"]
Nov 02 01:50:11.745093 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.745049    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.798446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.798411    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-dshm\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.798636 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.798477    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-tmp-dir\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.798727 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.798664    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ede2a-7f50-43fc-862b-b27ba373340d-tls-certs\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.798793 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.798742    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-home\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.798855 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.798797    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-model-cache\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.798855 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.798839    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzslm\" (UniqueName: \"kubernetes.io/projected/f24ede2a-7f50-43fc-862b-b27ba373340d-kube-api-access-qzslm\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.847945 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.847918    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-55bc84\"/\"openshift-service-ca.crt\""
Nov 02 01:50:11.854112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.854086    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-55bc84\"/\"distinct-w5-55bc84-kserve-self-signed-certs\""
Nov 02 01:50:11.854112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.854092    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-55bc84\"/\"default-dockercfg-m62zr\""
Nov 02 01:50:11.854263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.854129    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-55bc84\"/\"kube-root-ca.crt\""
Nov 02 01:50:11.858152 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.858129    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:11.858442 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.858424    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"]
Nov 02 01:50:11.900106 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.900045    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-home\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.900307 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.900162    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-model-cache\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.900307 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.900200    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzslm\" (UniqueName: \"kubernetes.io/projected/f24ede2a-7f50-43fc-862b-b27ba373340d-kube-api-access-qzslm\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.900307 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.900226    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-dshm\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.900307 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.900282    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-tmp-dir\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.900529 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.900394    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ede2a-7f50-43fc-862b-b27ba373340d-tls-certs\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.900529 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.900471    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-home\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.900598 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.900527    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-model-cache\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.900800 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.900779    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-tmp-dir\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.903160 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.903132    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-dshm\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.903310 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.903240    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ede2a-7f50-43fc-862b-b27ba373340d-tls-certs\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:11.961172 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:11.961139    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzslm\" (UniqueName: \"kubernetes.io/projected/f24ede2a-7f50-43fc-862b-b27ba373340d-kube-api-access-qzslm\") pod \"distinct-w5-55bc84-kserve-8fddcc5f7-kflkf\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") " pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:12.057873 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:12.057777    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:12.153164 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:12.148146    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:12.189894 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:50:12.189582    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:50:12.199189 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:12.197633    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:12.444736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:12.444710    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"]
Nov 02 01:50:12.444918 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:50:12.444898    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf24ede2a_7f50_43fc_862b_b27ba373340d.slice/crio-d96b328e40aa7fddb78a541d1054bcd72c986eb1e53471554d8bd7ee30067f49 WatchSource:0}: Error finding container d96b328e40aa7fddb78a541d1054bcd72c986eb1e53471554d8bd7ee30067f49: Status 404 returned error can't find the container with id d96b328e40aa7fddb78a541d1054bcd72c986eb1e53471554d8bd7ee30067f49
Nov 02 01:50:12.751519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:12.751425    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" event={"ID":"f24ede2a-7f50-43fc-862b-b27ba373340d","Type":"ContainerStarted","Data":"488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a"}
Nov 02 01:50:12.751519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:12.751480    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" event={"ID":"f24ede2a-7f50-43fc-862b-b27ba373340d","Type":"ContainerStarted","Data":"d96b328e40aa7fddb78a541d1054bcd72c986eb1e53471554d8bd7ee30067f49"}
Nov 02 01:50:12.763495 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:12.763470    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:50:12.873198 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:12.873150    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podStartSLOduration=1.873135856 podStartE2EDuration="1.873135856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:12.870447861 +0000 UTC m=+1105.775913219" watchObservedRunningTime="2026-11-02 01:50:12.873135856 +0000 UTC m=+1105.778601238"
Nov 02 01:50:13.058907 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:13.058827    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:15.879689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:15.879658    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:18.276927 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:18.276885    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:18.290211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:18.290185    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:18.807831 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:18.807803    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:50:20.751846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:20.751802    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:22.058683 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:22.058630    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:22.071944 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:22.071914    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:22.848996 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:22.848970    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:50:25.040655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:25.040613    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:25.561299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:25.561263    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:30.317219 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:30.317131    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:34.784068 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.784028    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"]
Nov 02 01:50:34.788034 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.788018    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:34.845641 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.845609    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"]
Nov 02 01:50:34.890460 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.890423    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ebdb3\"/\"distinct-w6-0ebdb3-kserve-self-signed-certs\""
Nov 02 01:50:34.921641 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.921602    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:34.967565 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.967498    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-home\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:34.967775 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.967635    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-model-cache\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:34.967775 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.967735    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-dshm\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:34.967901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.967778    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b841422-7581-414d-8ace-db272949f59d-tls-certs\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:34.967967 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.967938    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-tmp-dir\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:34.968102 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:34.968076    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zknbh\" (UniqueName: \"kubernetes.io/projected/4b841422-7581-414d-8ace-db272949f59d-kube-api-access-zknbh\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.069045 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.068941    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-tmp-dir\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.069237 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.069104    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknbh\" (UniqueName: \"kubernetes.io/projected/4b841422-7581-414d-8ace-db272949f59d-kube-api-access-zknbh\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.069237 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.069168    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-home\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.069237 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.069217    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-model-cache\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.069407 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.069247    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-dshm\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.069407 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.069273    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b841422-7581-414d-8ace-db272949f59d-tls-certs\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.069790 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.069765    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-home\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.072589 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.069875    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-model-cache\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.072589 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.070462    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-tmp-dir\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.072589 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.072461    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b841422-7581-414d-8ace-db272949f59d-tls-certs\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.073549 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.073530    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-dshm\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.130317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.130283    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zknbh\" (UniqueName: \"kubernetes.io/projected/4b841422-7581-414d-8ace-db272949f59d-kube-api-access-zknbh\") pod \"distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") " pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.145783 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.145750    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:35.401762 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.401730    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:35.784951 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.784919    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"]
Nov 02 01:50:35.786274 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:50:35.786243    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b841422_7581_414d_8ace_db272949f59d.slice/crio-593d3c08e21a1f8c5d740b8582dba0b0d7d2750193e252b7bb2e208f3eabcd59 WatchSource:0}: Error finding container 593d3c08e21a1f8c5d740b8582dba0b0d7d2750193e252b7bb2e208f3eabcd59: Status 404 returned error can't find the container with id 593d3c08e21a1f8c5d740b8582dba0b0d7d2750193e252b7bb2e208f3eabcd59
Nov 02 01:50:35.938859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.938826    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" event={"ID":"4b841422-7581-414d-8ace-db272949f59d","Type":"ContainerStarted","Data":"b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f"}
Nov 02 01:50:35.938859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:35.938865    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" event={"ID":"4b841422-7581-414d-8ace-db272949f59d","Type":"ContainerStarted","Data":"593d3c08e21a1f8c5d740b8582dba0b0d7d2750193e252b7bb2e208f3eabcd59"}
Nov 02 01:50:36.063042 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:36.062930    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podStartSLOduration=2.062909831 podStartE2EDuration="2.062909831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:36.059674987 +0000 UTC m=+1128.965140344" watchObservedRunningTime="2026-11-02 01:50:36.062909831 +0000 UTC m=+1128.968375189"
Nov 02 01:50:36.401805 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:36.401777    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:37.896525 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.896485    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"]
Nov 02 01:50:37.901781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.901757    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.920008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.919980    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.920259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.920040    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.920259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.920091    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx69b\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-kube-api-access-dx69b\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.920259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.920118    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c66bf0-994e-49a2-b485-d3dcff93447e-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.920259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.920137    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.920259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.920159    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.920259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.920193    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.920259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.920212    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.920775 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.920265    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:37.954270 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:37.954238    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-061fab-openshift-default-dockercfg-k8pns\""
Nov 02 01:50:38.013474 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.013443    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"]
Nov 02 01:50:38.021561 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.021521    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.021905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.021880    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.022006 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.021986    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.022108 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.022090    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.022221 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.022203    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.022295 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.022266    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx69b\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-kube-api-access-dx69b\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.022355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.022311    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c66bf0-994e-49a2-b485-d3dcff93447e-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.022407 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.022348    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.022407 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.022389    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.022845 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.022823    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.023551 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.023100    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.023675 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.023183    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.024262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.023432    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.024394 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.023865    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c66bf0-994e-49a2-b485-d3dcff93447e-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.026390 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.026354    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.042633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.042603    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.084822 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.084750    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.085167 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.085144    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx69b\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-kube-api-access-dx69b\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.220192 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.220110    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:38.696989 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.696952    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"]
Nov 02 01:50:38.701648 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:50:38.701621    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c66bf0_994e_49a2_b485_d3dcff93447e.slice/crio-80282909059eebafa6819bd1832a2f91137f86a8d283df53697264c8ed006820 WatchSource:0}: Error finding container 80282909059eebafa6819bd1832a2f91137f86a8d283df53697264c8ed006820: Status 404 returned error can't find the container with id 80282909059eebafa6819bd1832a2f91137f86a8d283df53697264c8ed006820
Nov 02 01:50:38.706010 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.705865    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:50:38.706010 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.705960    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:50:38.706010 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.706008    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:50:38.967606 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.967517    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" event={"ID":"73c66bf0-994e-49a2-b485-d3dcff93447e","Type":"ContainerStarted","Data":"a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0"}
Nov 02 01:50:38.967606 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:38.967588    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" event={"ID":"73c66bf0-994e-49a2-b485-d3dcff93447e","Type":"ContainerStarted","Data":"80282909059eebafa6819bd1832a2f91137f86a8d283df53697264c8ed006820"}
Nov 02 01:50:39.091496 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:39.091446    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podStartSLOduration=2.091432013 podStartE2EDuration="2.091432013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:39.088838426 +0000 UTC m=+1131.994303782" watchObservedRunningTime="2026-11-02 01:50:39.091432013 +0000 UTC m=+1131.996897370"
Nov 02 01:50:39.223319 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:39.223221    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:39.224888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:39.224855    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:50:39.225071 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:39.224917    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 01:50:40.094317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:40.094281    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:40.221465 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:40.221427    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:50:40.221650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:40.221486    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 01:50:41.220734 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:41.220697    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:50:41.221297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:41.220761    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 01:50:42.138267 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:50:42.138231    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:50:42.220566 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:42.220525    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:50:42.220741 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:42.220581    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 01:50:43.220401 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:43.220367    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:50:43.220841 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:43.220427    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 01:50:44.013071 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:44.013024    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:44.221341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:44.221311    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:50:44.221732 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:44.221377    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 01:50:44.978032 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:44.977989    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:45.226086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:45.223993    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:50:45.226086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:45.224116    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 01:50:45.402381 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:45.402342    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:45.415442 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:45.415415    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:46.042438 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:46.042411    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:50:46.225534 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:46.225504    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:46.225893 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:46.225871    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:46.226767 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:46.226751    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 01:50:49.750288 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:49.750248    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:54.547412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:54.547371    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:50:59.411948 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:50:59.411897    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:02.056422 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:02.056376    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:04.166615 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:04.166560    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:06.782001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:06.781966    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:08.950979 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:08.950943    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:12.140144 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:51:12.140109    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:51:13.778364 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:13.778329    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:18.531888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:18.531860    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:19.012292 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.012256    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"]
Nov 02 01:51:19.017575 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.017529    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.044051 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.044015    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-home\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.044225 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.044178    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-dshm\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.044290 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.044240    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-model-cache\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.044357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.044334    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvnr\" (UniqueName: \"kubernetes.io/projected/d3c07e88-8d03-43bb-b28f-e3ef7490775b-kube-api-access-hwvnr\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.044431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.044410    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tls-certs\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.044529 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.044515    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tmp-dir\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.075685 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.075645    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"]
Nov 02 01:51:19.119219 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.119183    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dbe849\"/\"distinct-2-w1-dbe849-kserve-self-signed-certs\""
Nov 02 01:51:19.122537 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.122511    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:19.146069 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.146025    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-model-cache\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.146246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.146224    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvnr\" (UniqueName: \"kubernetes.io/projected/d3c07e88-8d03-43bb-b28f-e3ef7490775b-kube-api-access-hwvnr\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.146352 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.146335    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tls-certs\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.146449 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.146424    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-model-cache\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.146552 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.146465    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tmp-dir\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.146642 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.146596    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-home\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.146730 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.146676    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-dshm\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.147048 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.147026    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tmp-dir\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.147309 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.147288    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-home\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.170845 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.170601    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-dshm\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.176461 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.172404    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tls-certs\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.207841 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.207811    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvnr\" (UniqueName: \"kubernetes.io/projected/d3c07e88-8d03-43bb-b28f-e3ef7490775b-kube-api-access-hwvnr\") pod \"distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") " pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.330683 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.330605    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:19.740588 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:51:19.740559    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3c07e88_8d03_43bb_b28f_e3ef7490775b.slice/crio-e930d258cc1c42635c61e338e718a4c06803ad93a0d53c1fe66790cda5caa96a WatchSource:0}: Error finding container e930d258cc1c42635c61e338e718a4c06803ad93a0d53c1fe66790cda5caa96a: Status 404 returned error can't find the container with id e930d258cc1c42635c61e338e718a4c06803ad93a0d53c1fe66790cda5caa96a
Nov 02 01:51:19.741013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:19.740991    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"]
Nov 02 01:51:20.335831 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:20.335791    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" event={"ID":"d3c07e88-8d03-43bb-b28f-e3ef7490775b","Type":"ContainerStarted","Data":"3688e2dfb5919339d31406bf7837cd00604f4e2d0da2e89b0c70b93ff0390164"}
Nov 02 01:51:20.336001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:20.335841    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" event={"ID":"d3c07e88-8d03-43bb-b28f-e3ef7490775b","Type":"ContainerStarted","Data":"e930d258cc1c42635c61e338e718a4c06803ad93a0d53c1fe66790cda5caa96a"}
Nov 02 01:51:20.457245 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:20.457189    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podStartSLOduration=2.457174411 podStartE2EDuration="2.457174411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:20.45480292 +0000 UTC m=+1173.360268277" watchObservedRunningTime="2026-11-02 01:51:20.457174411 +0000 UTC m=+1173.362639768"
Nov 02 01:51:21.331328 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:21.331296    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:23.344037 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:23.343997    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:28.140800 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:28.140756    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:29.330727 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:29.330694    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:29.344663 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:29.344635    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:29.418305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:29.418277    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:51:32.949013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:32.948975    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:37.765429 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:37.765398    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:42.137416 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:51:42.137378    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:51:42.551831 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.551738    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:42.861208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.861130    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"]
Nov 02 01:51:42.865438 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.865414    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:42.942538 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.942508    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-home\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:42.942706 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.942559    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tls-certs\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:42.942706 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.942670    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-dshm\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:42.942780 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.942703    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwb4\" (UniqueName: \"kubernetes.io/projected/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-kube-api-access-jlwb4\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:42.942815 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.942802    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tmp-dir\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:42.942846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.942839    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-model-cache\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:42.967845 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.967824    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"distinct-2-w3-061fab-kserve-self-signed-certs\""
Nov 02 01:51:42.971938 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.971919    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"]
Nov 02 01:51:42.980410 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:42.980383    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:43.044443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.044404    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-home\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.044666 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.044493    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tls-certs\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.044666 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.044546    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-dshm\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.044666 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.044569    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwb4\" (UniqueName: \"kubernetes.io/projected/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-kube-api-access-jlwb4\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.044666 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.044617    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tmp-dir\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.044875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.044675    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-model-cache\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.045133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.045106    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-home\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.045277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.045160    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tmp-dir\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.045277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.045209    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-model-cache\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.047226 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.047202    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tls-certs\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.047365 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.047301    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-dshm\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.106747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.106709    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwb4\" (UniqueName: \"kubernetes.io/projected/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-kube-api-access-jlwb4\") pod \"distinct-2-w3-061fab-kserve-98567c679-w5c5l\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") " pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.178744 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.178717    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:43.589581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:43.589550    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"]
Nov 02 01:51:43.590020 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:51:43.589996    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96254bb4_5c44_4d15_a6ab_19fd2d79ce62.slice/crio-ec52d4db2a57d81fdc489ccf87c0b01527af7efdce6af1626da7b615c93fa3cd WatchSource:0}: Error finding container ec52d4db2a57d81fdc489ccf87c0b01527af7efdce6af1626da7b615c93fa3cd: Status 404 returned error can't find the container with id ec52d4db2a57d81fdc489ccf87c0b01527af7efdce6af1626da7b615c93fa3cd
Nov 02 01:51:44.341498 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:44.341464    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:44.531912 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:44.531874    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" event={"ID":"96254bb4-5c44-4d15-a6ab-19fd2d79ce62","Type":"ContainerStarted","Data":"7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787"}
Nov 02 01:51:44.532109 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:44.531922    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" event={"ID":"96254bb4-5c44-4d15-a6ab-19fd2d79ce62","Type":"ContainerStarted","Data":"ec52d4db2a57d81fdc489ccf87c0b01527af7efdce6af1626da7b615c93fa3cd"}
Nov 02 01:51:44.649880 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:44.649765    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podStartSLOduration=2.649748251 podStartE2EDuration="2.649748251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:44.64924255 +0000 UTC m=+1197.554707907" watchObservedRunningTime="2026-11-02 01:51:44.649748251 +0000 UTC m=+1197.555213609"
Nov 02 01:51:45.179482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:45.179446    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:47.351178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:47.351137    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:47.523165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:47.523140    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:51:47.530321 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:47.530299    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:51:47.570949 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:47.570927    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:51:47.578343 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:47.578320    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:51:52.106459 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.106426    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:51:52.472339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.472303    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kfrdp-6qbtd"]
Nov 02 01:51:52.476826 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.476806    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.533518 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.533489    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kfrdp-6qbtd"]
Nov 02 01:51:52.559938 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.559908    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-trusted-ca-bundle\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.560182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.560155    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-service-ca-bundle\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.560270 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.560256    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1554bc18-c67c-47b6-bc69-d12bf3042835-archives-path\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.560389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.560374    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-885r2\" (UniqueName: \"kubernetes.io/projected/1554bc18-c67c-47b6-bc69-d12bf3042835-kube-api-access-885r2\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.578193 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.578167    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8qw44\""
Nov 02 01:51:52.661782 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.661747    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-885r2\" (UniqueName: \"kubernetes.io/projected/1554bc18-c67c-47b6-bc69-d12bf3042835-kube-api-access-885r2\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.661987 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.661790    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-trusted-ca-bundle\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.661987 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.661940    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-service-ca-bundle\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.662172 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.661999    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1554bc18-c67c-47b6-bc69-d12bf3042835-archives-path\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.662424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.662401    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1554bc18-c67c-47b6-bc69-d12bf3042835-archives-path\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.662631 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.662610    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-trusted-ca-bundle\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.662719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.662700    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-service-ca-bundle\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.722958 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.722881    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-885r2\" (UniqueName: \"kubernetes.io/projected/1554bc18-c67c-47b6-bc69-d12bf3042835-kube-api-access-885r2\") pod \"periodic-gathering-kfrdp-6qbtd\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") " pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:52.790484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:52.790449    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:51:53.179574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:53.179539    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:53.189084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:53.189041    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kfrdp-6qbtd"]
Nov 02 01:51:53.193485 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:53.193463    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:53.607297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:53.607209    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfrdp-6qbtd" event={"ID":"1554bc18-c67c-47b6-bc69-d12bf3042835","Type":"ContainerStarted","Data":"deb360c3d5a1e6171fcaa35c1b02848ff3784fccf3edbd5adade4dbb3f57fcd4"}
Nov 02 01:51:53.607297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:53.607272    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfrdp-6qbtd" event={"ID":"1554bc18-c67c-47b6-bc69-d12bf3042835","Type":"ContainerStarted","Data":"ddaf5e43276af0bb43ca2813942ec1c9db33b5e288e160cfbc7c0ca42a097dce"}
Nov 02 01:51:53.619368 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:53.619335    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:51:53.725672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:53.725613    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-kfrdp-6qbtd" podStartSLOduration=1.7255932029999999 podStartE2EDuration="1.725593203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:53.721895852 +0000 UTC m=+1206.627361213" watchObservedRunningTime="2026-11-02 01:51:53.725593203 +0000 UTC m=+1206.631058562"
Nov 02 01:51:56.969948 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:51:56.969912    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:01.747333 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:01.747292    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:06.553859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:06.553820    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:11.344036 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:11.343993    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:12.138025 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:52:12.137996    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:52:16.151832 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:16.151790    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:20.939018 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:20.938973    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:23.707768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.707734    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"]
Nov 02 01:52:23.723361 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.723322    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.832807 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.832768    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"]
Nov 02 01:52:23.887095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.887016    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.887309 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.887109    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.887309 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.887203    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d28689-caf9-4713-b737-c581416b28a9-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.887445 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.887323    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.887445 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.887350    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.887445 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.887391    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp58j\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-kube-api-access-gp58j\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.887445 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.887419    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.887678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.887479    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.887678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.887540    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d28689-caf9-4713-b737-c581416b28a9-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.991823 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.991734    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.991823 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.991774    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.991823 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.991805    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp58j\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-kube-api-access-gp58j\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.992157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.991831    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.992157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.991873    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.992157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.991909    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d28689-caf9-4713-b737-c581416b28a9-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.992157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.991961    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.992157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.992002    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.992157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.992141    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.992536 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.992182    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d28689-caf9-4713-b737-c581416b28a9-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.992827 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.992800    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.992940 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.992829    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.993091 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.993068    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.993435 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.993411    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d28689-caf9-4713-b737-c581416b28a9-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.996095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.995431    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:23.996095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:23.995871    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d28689-caf9-4713-b737-c581416b28a9-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:24.056397 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:24.056314    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp58j\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-kube-api-access-gp58j\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:24.056397 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:24.057006    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:24.190814 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:24.190772    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:24.338205 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:24.338127    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:24.811855 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:52:24.811826    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d28689_caf9_4713_b737_c581416b28a9.slice/crio-e90f97792a6029468cc99c3e2b245ed9f1536b7a05c8bf3ec8651c370f693963 WatchSource:0}: Error finding container e90f97792a6029468cc99c3e2b245ed9f1536b7a05c8bf3ec8651c370f693963: Status 404 returned error can't find the container with id e90f97792a6029468cc99c3e2b245ed9f1536b7a05c8bf3ec8651c370f693963
Nov 02 01:52:24.812814 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:24.812760    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"]
Nov 02 01:52:24.815512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:24.815473    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:52:24.815749 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:24.815724    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:52:24.815899 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:24.815881    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:52:24.911279 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:24.911226    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" event={"ID":"d2d28689-caf9-4713-b737-c581416b28a9","Type":"ContainerStarted","Data":"e90f97792a6029468cc99c3e2b245ed9f1536b7a05c8bf3ec8651c370f693963"}
Nov 02 01:52:25.783085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:25.778336    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:25.921118 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:25.921049    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" event={"ID":"d2d28689-caf9-4713-b737-c581416b28a9","Type":"ContainerStarted","Data":"c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4"}
Nov 02 01:52:26.107903 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:26.107807    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" podStartSLOduration=3.107793674 podStartE2EDuration="3.107793674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:26.103440642 +0000 UTC m=+1239.008905999" watchObservedRunningTime="2026-11-02 01:52:26.107793674 +0000 UTC m=+1239.013259030"
Nov 02 01:52:26.339339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:26.339310    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:26.341216 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:26.341192    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:52:26.341340 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:26.341245    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 01:52:27.339327 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:27.339294    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:52:27.339910 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:27.339348    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 01:52:28.339465 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:28.339425    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:52:28.339882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:28.339480    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 01:52:29.339029 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:29.338999    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:52:29.339237 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:29.339072    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 01:52:29.727832 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:29.727725    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:30.338732 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:30.338691    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:52:30.338917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:30.338756    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 01:52:30.683387 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:30.683327    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:31.338516 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:31.338477    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:52:31.338946 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:31.338534    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 01:52:32.500454 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:32.500407    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:32.501263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:32.501237    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:32.502608 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:32.502586    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 01:52:35.445210 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.445173    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:35.569523 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.569487    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"]
Nov 02 01:52:35.573703 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.573681    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.631121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.631088    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"]
Nov 02 01:52:35.675169 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.675135    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:52:35.675341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.675283    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:52:35.705214 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.705134    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9np\" (UniqueName: \"kubernetes.io/projected/a96772f5-05c8-49f7-9c31-1fccda98f0f7-kube-api-access-qv9np\") pod \"maas-api-revoke-sub-3025d6f1-2pkxh\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.705357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.705278    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3025d6f1-2pkxh\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.705357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.705334    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3025d6f1-2pkxh\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.806195 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.806161    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9np\" (UniqueName: \"kubernetes.io/projected/a96772f5-05c8-49f7-9c31-1fccda98f0f7-kube-api-access-qv9np\") pod \"maas-api-revoke-sub-3025d6f1-2pkxh\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.806387 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.806252    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3025d6f1-2pkxh\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.806387 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.806306    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3025d6f1-2pkxh\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.806972 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.806949    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3025d6f1-2pkxh\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.808763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.808743    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3025d6f1-2pkxh\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.866891 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.866864    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9np\" (UniqueName: \"kubernetes.io/projected/a96772f5-05c8-49f7-9c31-1fccda98f0f7-kube-api-access-qv9np\") pod \"maas-api-revoke-sub-3025d6f1-2pkxh\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:35.888862 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:35.888839    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:36.271677 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:36.271647    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"]
Nov 02 01:52:36.271835 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:52:36.271767    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96772f5_05c8_49f7_9c31_1fccda98f0f7.slice/crio-e67117410b213545234e28985e7f0c2453b15e5736611c70ea3fde422f3983a9 WatchSource:0}: Error finding container e67117410b213545234e28985e7f0c2453b15e5736611c70ea3fde422f3983a9: Status 404 returned error can't find the container with id e67117410b213545234e28985e7f0c2453b15e5736611c70ea3fde422f3983a9
Nov 02 01:52:37.036457 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:37.036417    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh" event={"ID":"a96772f5-05c8-49f7-9c31-1fccda98f0f7","Type":"ContainerStarted","Data":"cef463de35f05a3e7d8824f2ba951a3d462599cf971705436d17f174d3978b4f"}
Nov 02 01:52:37.036957 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:37.036500    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh" event={"ID":"a96772f5-05c8-49f7-9c31-1fccda98f0f7","Type":"ContainerStarted","Data":"e67117410b213545234e28985e7f0c2453b15e5736611c70ea3fde422f3983a9"}
Nov 02 01:52:37.151556 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:37.150558    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh" podStartSLOduration=2.15054243 podStartE2EDuration="2.15054243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:37.150175163 +0000 UTC m=+1250.055640523" watchObservedRunningTime="2026-11-02 01:52:37.15054243 +0000 UTC m=+1250.056007788"
Nov 02 01:52:38.048943 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:38.048911    4912 generic.go:356] "Generic (PLEG): container finished" podID="a96772f5-05c8-49f7-9c31-1fccda98f0f7" containerID="cef463de35f05a3e7d8824f2ba951a3d462599cf971705436d17f174d3978b4f" exitCode=0
Nov 02 01:52:38.049357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:38.048997    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh" event={"ID":"a96772f5-05c8-49f7-9c31-1fccda98f0f7","Type":"ContainerDied","Data":"cef463de35f05a3e7d8824f2ba951a3d462599cf971705436d17f174d3978b4f"}
Nov 02 01:52:39.220510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.220483    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:39.349405 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.349332    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-xks-ca\") pod \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") "
Nov 02 01:52:39.349405 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.349386    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-openshift-ca\") pod \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") "
Nov 02 01:52:39.349590 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.349414    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96772f5-05c8-49f7-9c31-1fccda98f0f7-kube-api-access-qv9np\" (UniqueName: \"kubernetes.io/projected/a96772f5-05c8-49f7-9c31-1fccda98f0f7-kube-api-access-qv9np\") pod \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\" (UID: \"a96772f5-05c8-49f7-9c31-1fccda98f0f7\") "
Nov 02 01:52:39.349711 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.349691    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-openshift-ca" pod "a96772f5-05c8-49f7-9c31-1fccda98f0f7" (UID: "a96772f5-05c8-49f7-9c31-1fccda98f0f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:52:39.349804 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.349792    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:39.351595 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.351567    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-xks-ca" pod "a96772f5-05c8-49f7-9c31-1fccda98f0f7" (UID: "a96772f5-05c8-49f7-9c31-1fccda98f0f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:52:39.351715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.351634    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96772f5-05c8-49f7-9c31-1fccda98f0f7-kube-api-access-qv9np" pod "a96772f5-05c8-49f7-9c31-1fccda98f0f7" (UID: "a96772f5-05c8-49f7-9c31-1fccda98f0f7"). InnerVolumeSpecName "kube-api-access-qv9np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:39.451367 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.451332    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96772f5-05c8-49f7-9c31-1fccda98f0f7-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:39.451367 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.451362    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv9np\" (UniqueName: \"kubernetes.io/projected/a96772f5-05c8-49f7-9c31-1fccda98f0f7-kube-api-access-qv9np\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:39.710946 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:39.710910    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:40.066760 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:40.066685    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh" event={"ID":"a96772f5-05c8-49f7-9c31-1fccda98f0f7","Type":"ContainerDied","Data":"e67117410b213545234e28985e7f0c2453b15e5736611c70ea3fde422f3983a9"}
Nov 02 01:52:40.066760 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:40.066733    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e67117410b213545234e28985e7f0c2453b15e5736611c70ea3fde422f3983a9"
Nov 02 01:52:40.066943 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:40.066785    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"
Nov 02 01:52:40.262128 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:40.262095    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:42.138723 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:52:42.138686    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:52:44.694889 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:44.694851    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:45.059852 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.059742    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:45.182611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.182574    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"]
Nov 02 01:52:45.183262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.183244    4912 cpu_manager.go:414] "containerMap: removing container" podUID="a96772f5-05c8-49f7-9c31-1fccda98f0f7" containerName="revoke-keys"
Nov 02 01:52:45.183262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.183261    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96772f5-05c8-49f7-9c31-1fccda98f0f7" containerName="revoke-keys"
Nov 02 01:52:45.183436 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.183374    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96772f5-05c8-49f7-9c31-1fccda98f0f7" containerName="revoke-keys"
Nov 02 01:52:45.187150 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.187132    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.288759 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.288713    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed0600\"/\"default-dockercfg-qr9gk\""
Nov 02 01:52:45.288991 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.288767    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ed0600\"/\"kube-root-ca.crt\""
Nov 02 01:52:45.289144 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.289126    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ed0600\"/\"openshift-service-ca.crt\""
Nov 02 01:52:45.289210 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.289163    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed0600\"/\"distinct-2-w4-ed0600-kserve-self-signed-certs\""
Nov 02 01:52:45.294133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.294109    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"]
Nov 02 01:52:45.298447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.298418    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:45.332700 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.332597    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tmp-dir\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.332876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.332780    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlwvt\" (UniqueName: \"kubernetes.io/projected/b6226fcd-61ac-4173-a1fa-28378df2bb1b-kube-api-access-hlwvt\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.332876 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.332808    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-model-cache\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.332991 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.332885    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tls-certs\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.332991 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.332908    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-home\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.332991 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.332952    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-dshm\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.433921 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.433881    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwvt\" (UniqueName: \"kubernetes.io/projected/b6226fcd-61ac-4173-a1fa-28378df2bb1b-kube-api-access-hlwvt\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.433921 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.433917    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-model-cache\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.434232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.433982    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tls-certs\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.434232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.434005    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-home\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.434232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.434033    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-dshm\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.434232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.434102    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tmp-dir\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.434455 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.434378    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-model-cache\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.434515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.434499    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tmp-dir\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.434602 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.434580    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-home\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.436523 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.436501    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-dshm\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.436616 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.436572    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tls-certs\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.495316 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.495281    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwvt\" (UniqueName: \"kubernetes.io/projected/b6226fcd-61ac-4173-a1fa-28378df2bb1b-kube-api-access-hlwvt\") pod \"distinct-2-w4-ed0600-kserve-76c7999d9-njm6f\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") " pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.501166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.501143    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:45.916378 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:45.916343    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"]
Nov 02 01:52:45.916812 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:52:45.916748    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6226fcd_61ac_4173_a1fa_28378df2bb1b.slice/crio-721e236cbf2260206088f6b844cff08c1c0ea15ecb6b602b8d3ad42f5d66364f WatchSource:0}: Error finding container 721e236cbf2260206088f6b844cff08c1c0ea15ecb6b602b8d3ad42f5d66364f: Status 404 returned error can't find the container with id 721e236cbf2260206088f6b844cff08c1c0ea15ecb6b602b8d3ad42f5d66364f
Nov 02 01:52:46.128508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:46.128464    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" event={"ID":"b6226fcd-61ac-4173-a1fa-28378df2bb1b","Type":"ContainerStarted","Data":"e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9"}
Nov 02 01:52:46.128508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:46.128516    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" event={"ID":"b6226fcd-61ac-4173-a1fa-28378df2bb1b","Type":"ContainerStarted","Data":"721e236cbf2260206088f6b844cff08c1c0ea15ecb6b602b8d3ad42f5d66364f"}
Nov 02 01:52:46.248191 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:46.248093    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podStartSLOduration=1.248078299 podStartE2EDuration="1.248078299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:46.244312106 +0000 UTC m=+1259.149777462" watchObservedRunningTime="2026-11-02 01:52:46.248078299 +0000 UTC m=+1259.153543660"
Nov 02 01:52:46.502120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:46.501994    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:49.978962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:49.978924    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:52.352026 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:52.351986    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:54.857429 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:54.857388    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:55.501956 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:55.501924    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:55.514933 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:55.514906    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:56.225436 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:56.225407    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:52:57.920094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:57.920024    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:52:59.677717 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:52:59.677679    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:01.175905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.175873    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"]
Nov 02 01:53:01.181263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.181241    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.238149 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.238118    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"]
Nov 02 01:53:01.245024 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.244998    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a50abcf4-7h78b\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.245202 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.245142    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a50abcf4-7h78b\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.245202 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.245173    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmf4w\" (UniqueName: \"kubernetes.io/projected/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-kube-api-access-cmf4w\") pod \"maas-api-revoke-sub-a50abcf4-7h78b\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.265456 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.265390    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:01.265456 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.265404    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:01.346428 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.346391    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a50abcf4-7h78b\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.346631 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.346445    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmf4w\" (UniqueName: \"kubernetes.io/projected/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-kube-api-access-cmf4w\") pod \"maas-api-revoke-sub-a50abcf4-7h78b\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.346631 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.346586    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a50abcf4-7h78b\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.347040 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.347020    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a50abcf4-7h78b\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.349146 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.349127    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a50abcf4-7h78b\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.407522 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.407492    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmf4w\" (UniqueName: \"kubernetes.io/projected/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-kube-api-access-cmf4w\") pod \"maas-api-revoke-sub-a50abcf4-7h78b\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.506794 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.506717    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:01.897369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:01.897337    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"]
Nov 02 01:53:01.898250 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:01.898226    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a8aac8_11fe_4c9e_9c01_e472fae4f09c.slice/crio-0dc93a7a2c086ac04b420485ee8ddd928a7385d0891e3316121cfcaff2234558 WatchSource:0}: Error finding container 0dc93a7a2c086ac04b420485ee8ddd928a7385d0891e3316121cfcaff2234558: Status 404 returned error can't find the container with id 0dc93a7a2c086ac04b420485ee8ddd928a7385d0891e3316121cfcaff2234558
Nov 02 01:53:02.264512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:02.264469    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b" event={"ID":"01a8aac8-11fe-4c9e-9c01-e472fae4f09c","Type":"ContainerStarted","Data":"9d00b5ace4902157908328597433c1cb11efd17f79656b0348dfb848e2017b5d"}
Nov 02 01:53:02.264944 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:02.264539    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b" event={"ID":"01a8aac8-11fe-4c9e-9c01-e472fae4f09c","Type":"ContainerStarted","Data":"0dc93a7a2c086ac04b420485ee8ddd928a7385d0891e3316121cfcaff2234558"}
Nov 02 01:53:02.386318 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:02.386258    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b" podStartSLOduration=1.386243848 podStartE2EDuration="1.386243848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:02.382129999 +0000 UTC m=+1275.287595355" watchObservedRunningTime="2026-11-02 01:53:02.386243848 +0000 UTC m=+1275.291709245"
Nov 02 01:53:03.273851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:03.273820    4912 generic.go:356] "Generic (PLEG): container finished" podID="01a8aac8-11fe-4c9e-9c01-e472fae4f09c" containerID="9d00b5ace4902157908328597433c1cb11efd17f79656b0348dfb848e2017b5d" exitCode=0
Nov 02 01:53:03.274288 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:03.273877    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b" event={"ID":"01a8aac8-11fe-4c9e-9c01-e472fae4f09c","Type":"ContainerDied","Data":"9d00b5ace4902157908328597433c1cb11efd17f79656b0348dfb848e2017b5d"}
Nov 02 01:53:04.440995 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.440969    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:04.463304 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.463263    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:04.588508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.588432    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-kube-api-access-cmf4w\" (UniqueName: \"kubernetes.io/projected/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-kube-api-access-cmf4w\") pod \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") "
Nov 02 01:53:04.588508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.588505    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-openshift-ca\") pod \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") "
Nov 02 01:53:04.588742 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.588553    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-xks-ca\") pod \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\" (UID: \"01a8aac8-11fe-4c9e-9c01-e472fae4f09c\") "
Nov 02 01:53:04.588926 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.588884    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-openshift-ca" pod "01a8aac8-11fe-4c9e-9c01-e472fae4f09c" (UID: "01a8aac8-11fe-4c9e-9c01-e472fae4f09c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:04.589160 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.589139    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:04.590851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.590824    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-kube-api-access-cmf4w" pod "01a8aac8-11fe-4c9e-9c01-e472fae4f09c" (UID: "01a8aac8-11fe-4c9e-9c01-e472fae4f09c"). InnerVolumeSpecName "kube-api-access-cmf4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:04.590953 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.590874    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-xks-ca" pod "01a8aac8-11fe-4c9e-9c01-e472fae4f09c" (UID: "01a8aac8-11fe-4c9e-9c01-e472fae4f09c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:04.701126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.701087    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmf4w\" (UniqueName: \"kubernetes.io/projected/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-kube-api-access-cmf4w\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:04.701126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:04.701121    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a8aac8-11fe-4c9e-9c01-e472fae4f09c-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:05.296149 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:05.296119    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"
Nov 02 01:53:05.296352 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:05.296178    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b" event={"ID":"01a8aac8-11fe-4c9e-9c01-e472fae4f09c","Type":"ContainerDied","Data":"0dc93a7a2c086ac04b420485ee8ddd928a7385d0891e3316121cfcaff2234558"}
Nov 02 01:53:05.296352 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:05.296223    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dc93a7a2c086ac04b420485ee8ddd928a7385d0891e3316121cfcaff2234558"
Nov 02 01:53:08.424232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.424197    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"]
Nov 02 01:53:08.425747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.425720    4912 cpu_manager.go:414] "containerMap: removing container" podUID="01a8aac8-11fe-4c9e-9c01-e472fae4f09c" containerName="revoke-keys"
Nov 02 01:53:08.425908 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.425890    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a8aac8-11fe-4c9e-9c01-e472fae4f09c" containerName="revoke-keys"
Nov 02 01:53:08.426243 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.426226    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a8aac8-11fe-4c9e-9c01-e472fae4f09c" containerName="revoke-keys"
Nov 02 01:53:08.433327 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.433304    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.533083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.505463    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shsfl\" (UniqueName: \"kubernetes.io/projected/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-kube-api-access-shsfl\") pod \"maas-api-revoke-sub-afbf6de3-9wgxb\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.533083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.505628    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afbf6de3-9wgxb\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.533083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.505669    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afbf6de3-9wgxb\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.548436 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.548409    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:08.548911 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.548778    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:08.612634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.612579    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shsfl\" (UniqueName: \"kubernetes.io/projected/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-kube-api-access-shsfl\") pod \"maas-api-revoke-sub-afbf6de3-9wgxb\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.613130 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.613101    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afbf6de3-9wgxb\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.613336 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.613316    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afbf6de3-9wgxb\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.633086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.630597    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afbf6de3-9wgxb\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.644288 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.644212    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afbf6de3-9wgxb\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.656148 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.656111    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"]
Nov 02 01:53:08.715138 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.714242    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shsfl\" (UniqueName: \"kubernetes.io/projected/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-kube-api-access-shsfl\") pod \"maas-api-revoke-sub-afbf6de3-9wgxb\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:08.847625 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:08.847391    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:09.342937 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.342900    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"]
Nov 02 01:53:09.347884 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:09.347846    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65f2d2f_d1e8_4497_b4be_885b48ddaf2c.slice/crio-3ce8a7d055e15782d4dd2c063a2ff55c946a8ca26b0694fdf46cf4645d55da55 WatchSource:0}: Error finding container 3ce8a7d055e15782d4dd2c063a2ff55c946a8ca26b0694fdf46cf4645d55da55: Status 404 returned error can't find the container with id 3ce8a7d055e15782d4dd2c063a2ff55c946a8ca26b0694fdf46cf4645d55da55
Nov 02 01:53:09.731673 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.731635    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"]
Nov 02 01:53:09.737528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.737508    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.789962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.789924    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c62671-openshift-default-dockercfg-nw98n\""
Nov 02 01:53:09.838988 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.838898    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f4fe9-2bb4-4324-9951-976238151431-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.838988 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.838978    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lqkr\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-kube-api-access-7lqkr\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.839239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.839005    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.839239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.839110    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.839239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.839150    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.839239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.839209    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.839239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.839237    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.839484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.839297    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f4fe9-2bb4-4324-9951-976238151431-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.839484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.839349    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.847779 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.847749    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"]
Nov 02 01:53:09.940444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.940411    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f4fe9-2bb4-4324-9951-976238151431-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.940664 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.940494    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqkr\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-kube-api-access-7lqkr\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.940664 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.940514    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.940664 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.940587    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.940664 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.940619    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.940895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.940692    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.940895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.940720    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.940895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.940748    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f4fe9-2bb4-4324-9951-976238151431-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.940895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.940813    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.941120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.941101    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.941312 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.941266    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.941381 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.941326    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f4fe9-2bb4-4324-9951-976238151431-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.941381 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.941327    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.941600 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.941531    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.943738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.943693    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:09.944411 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:09.944366    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f4fe9-2bb4-4324-9951-976238151431-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:10.002850 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.002766    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lqkr\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-kube-api-access-7lqkr\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:10.002850 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.002820    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:10.053955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.053922    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:10.106133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.106049    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:10.383260 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.383215    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb" event={"ID":"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c","Type":"ContainerStarted","Data":"7d7c10823c2828fae62e8548871063ac644e5799d7f9f4768515347d3f99a1a2"}
Nov 02 01:53:10.383447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.383282    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb" event={"ID":"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c","Type":"ContainerStarted","Data":"3ce8a7d055e15782d4dd2c063a2ff55c946a8ca26b0694fdf46cf4645d55da55"}
Nov 02 01:53:10.501821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.501789    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"]
Nov 02 01:53:10.503206 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:10.503174    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532f4fe9_2bb4_4324_9951_976238151431.slice/crio-76bf89a5a913beb1ec1151b60fda47a5d695275f1a24b0cfe668ff7afbc594b9 WatchSource:0}: Error finding container 76bf89a5a913beb1ec1151b60fda47a5d695275f1a24b0cfe668ff7afbc594b9: Status 404 returned error can't find the container with id 76bf89a5a913beb1ec1151b60fda47a5d695275f1a24b0cfe668ff7afbc594b9
Nov 02 01:53:10.506187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.506156    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:10.506272 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.506221    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:10.506272 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:10.506250    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:11.395116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:11.395083    4912 generic.go:356] "Generic (PLEG): container finished" podID="b65f2d2f-d1e8-4497-b4be-885b48ddaf2c" containerID="7d7c10823c2828fae62e8548871063ac644e5799d7f9f4768515347d3f99a1a2" exitCode=0
Nov 02 01:53:11.395630 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:11.395170    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb" event={"ID":"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c","Type":"ContainerDied","Data":"7d7c10823c2828fae62e8548871063ac644e5799d7f9f4768515347d3f99a1a2"}
Nov 02 01:53:11.396986 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:11.396964    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" event={"ID":"532f4fe9-2bb4-4324-9951-976238151431","Type":"ContainerStarted","Data":"542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315"}
Nov 02 01:53:11.397126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:11.397004    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" event={"ID":"532f4fe9-2bb4-4324-9951-976238151431","Type":"ContainerStarted","Data":"76bf89a5a913beb1ec1151b60fda47a5d695275f1a24b0cfe668ff7afbc594b9"}
Nov 02 01:53:11.640485 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:11.640434    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" podStartSLOduration=2.640420903 podStartE2EDuration="2.640420903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:11.635869414 +0000 UTC m=+1284.541334772" watchObservedRunningTime="2026-11-02 01:53:11.640420903 +0000 UTC m=+1284.545886333"
Nov 02 01:53:12.076116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.073559    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:12.076116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.075148    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 01:53:12.076116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.075198    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 01:53:12.138156 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:53:12.138121    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:53:12.604959 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.604926    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:12.706324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.706244    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-openshift-ca\") pod \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") "
Nov 02 01:53:12.706468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.706352    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-kube-api-access-shsfl\" (UniqueName: \"kubernetes.io/projected/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-kube-api-access-shsfl\") pod \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") "
Nov 02 01:53:12.706542 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.706471    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-xks-ca\") pod \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\" (UID: \"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c\") "
Nov 02 01:53:12.706637 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.706611    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-openshift-ca" pod "b65f2d2f-d1e8-4497-b4be-885b48ddaf2c" (UID: "b65f2d2f-d1e8-4497-b4be-885b48ddaf2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:12.706994 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.706970    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:12.709406 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.709376    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-xks-ca" pod "b65f2d2f-d1e8-4497-b4be-885b48ddaf2c" (UID: "b65f2d2f-d1e8-4497-b4be-885b48ddaf2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:12.709507 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.709472    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-kube-api-access-shsfl" pod "b65f2d2f-d1e8-4497-b4be-885b48ddaf2c" (UID: "b65f2d2f-d1e8-4497-b4be-885b48ddaf2c"). InnerVolumeSpecName "kube-api-access-shsfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:12.807964 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.807923    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:12.807964 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:12.807958    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shsfl\" (UniqueName: \"kubernetes.io/projected/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c-kube-api-access-shsfl\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:13.055530 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:13.055440    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 01:53:13.055791 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:13.055762    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 01:53:13.422348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:13.422321    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"
Nov 02 01:53:13.422496 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:13.422358    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb" event={"ID":"b65f2d2f-d1e8-4497-b4be-885b48ddaf2c","Type":"ContainerDied","Data":"3ce8a7d055e15782d4dd2c063a2ff55c946a8ca26b0694fdf46cf4645d55da55"}
Nov 02 01:53:13.422496 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:13.422419    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ce8a7d055e15782d4dd2c063a2ff55c946a8ca26b0694fdf46cf4645d55da55"
Nov 02 01:53:14.064082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:14.060425    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 01:53:14.064082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:14.060492    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 01:53:14.966652 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:14.962142    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:15.073083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:15.070159    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 01:53:15.073083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:15.070223    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 01:53:15.422506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:15.422469    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:15.662081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:15.658672    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:16.066084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:16.057472    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 01:53:16.066084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:16.057549    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 01:53:17.055177 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:17.055139    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 01:53:17.055685 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:17.055215    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 01:53:18.058811 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:18.058785    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:18.059268 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:18.059151    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:18.059797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:18.059779    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 01:53:19.754982 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:19.754951    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:22.944486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:22.944445    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"]
Nov 02 01:53:22.945285 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:22.945265    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b65f2d2f-d1e8-4497-b4be-885b48ddaf2c" containerName="revoke-keys"
Nov 02 01:53:22.945285 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:22.945286    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65f2d2f-d1e8-4497-b4be-885b48ddaf2c" containerName="revoke-keys"
Nov 02 01:53:22.945545 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:22.945462    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65f2d2f-d1e8-4497-b4be-885b48ddaf2c" containerName="revoke-keys"
Nov 02 01:53:22.950320 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:22.950295    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.034962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.034931    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.035274 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.035251    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.035371 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.035291    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.035467 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.035423    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.035640 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.035597    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.035640 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.035630    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.035771 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.035686    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.035771 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.035766    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5gz\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-kube-api-access-cq5gz\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.035883 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.035822    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137242 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137204    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137291    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137320    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137350    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137435    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137462    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137484    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137536    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5gz\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-kube-api-access-cq5gz\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137607    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137679    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.137856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.137789    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.138141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.138117    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.138249 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.138165    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.138311 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.138289    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.140005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.139983    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.140201 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.140182    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.168269 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.168239    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"]
Nov 02 01:53:23.198334 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.198277    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.198568 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.198545    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5gz\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-kube-api-access-cq5gz\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.264240 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.264212    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:23.698885 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:23.698856    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94764ea2_8bb7_4ce7_897a_fd0a1886f97f.slice/crio-a111fc89376ff5e280b21605af3ddf2cc4330bc35aebba078d505c4f4f82458e WatchSource:0}: Error finding container a111fc89376ff5e280b21605af3ddf2cc4330bc35aebba078d505c4f4f82458e: Status 404 returned error can't find the container with id a111fc89376ff5e280b21605af3ddf2cc4330bc35aebba078d505c4f4f82458e
Nov 02 01:53:23.699814 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.699790    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"]
Nov 02 01:53:23.701395 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.701365    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:23.701476 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.701424    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:23.701476 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:23.701455    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:24.525694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.525662    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" event={"ID":"94764ea2-8bb7-4ce7-897a-fd0a1886f97f","Type":"ContainerStarted","Data":"2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9"}
Nov 02 01:53:24.526215 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.525720    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" event={"ID":"94764ea2-8bb7-4ce7-897a-fd0a1886f97f","Type":"ContainerStarted","Data":"a111fc89376ff5e280b21605af3ddf2cc4330bc35aebba078d505c4f4f82458e"}
Nov 02 01:53:24.579468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.579421    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:24.582947 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.582925    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"]
Nov 02 01:53:24.587228 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.587209    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.685589 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.685551    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.686888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.686861    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwrd\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-kube-api-access-dhwrd\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.691515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.690361    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.691515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.690487    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.691515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.690561    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.691515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.690635    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.691515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.690674    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.691515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.690704    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.691515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.690782    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.691515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.690188    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"]
Nov 02 01:53:24.735080 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.731645    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" podStartSLOduration=2.731625388 podStartE2EDuration="2.731625388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:24.728534168 +0000 UTC m=+1297.633999526" watchObservedRunningTime="2026-11-02 01:53:24.731625388 +0000 UTC m=+1297.637090744"
Nov 02 01:53:24.791955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.791825    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwrd\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-kube-api-access-dhwrd\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.791955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.791911    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.792510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.791998    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.792510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.792080    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.792510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.792136    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.792510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.792173    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.792510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.792209    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.792510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.792263    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.792510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.792360    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.793723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.793217    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.794017 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.793953    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.795925 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.795851    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.796161 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.796138    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.798271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.798225    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.798452 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.798421    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.803887 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.803867    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.888087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.875842    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwrd\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-kube-api-access-dhwrd\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.888087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.876308    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:24.903720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:24.902462    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:25.086133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.085182    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"]
Nov 02 01:53:25.091890 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.091862    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2457cd74-7gj89"]
Nov 02 01:53:25.226526 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.226493    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d9d7ca-3c71-49b8-babb-2d08e6e49221" path="/var/lib/kubelet/pods/b6d9d7ca-3c71-49b8-babb-2d08e6e49221/volumes"
Nov 02 01:53:25.272149 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.272115    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:25.276142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.272460    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 01:53:25.276142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.272512    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 01:53:25.389143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.387868    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"]
Nov 02 01:53:25.400916 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.400866    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:25.401044 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.400959    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:25.401044 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.401001    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:25.535355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.535329    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" event={"ID":"5e8cfca1-a503-44b7-a3d2-d428da9374aa","Type":"ContainerStarted","Data":"29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec"}
Nov 02 01:53:25.535831 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.535810    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" event={"ID":"5e8cfca1-a503-44b7-a3d2-d428da9374aa","Type":"ContainerStarted","Data":"08b9f7822000769dfac8395ac43b7ef977903cf9b69f904091df9534fe95dbba"}
Nov 02 01:53:25.674444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.674336    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podStartSLOduration=1.6743164720000001 podStartE2EDuration="1.674316472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:25.658798618 +0000 UTC m=+1298.564263973" watchObservedRunningTime="2026-11-02 01:53:25.674316472 +0000 UTC m=+1298.579781828"
Nov 02 01:53:25.904092 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:25.904039    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:26.265654 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:26.265619    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 01:53:26.265994 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:26.265941    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 01:53:26.906214 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:26.906157    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 01:53:26.906735 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:26.906240    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 01:53:27.274176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:27.266488    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 01:53:27.274176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:27.266549    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 01:53:27.904145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:27.903941    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 01:53:27.904145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:27.903999    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 01:53:28.269164 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:28.266445    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 01:53:28.269164 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:28.266504    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 01:53:28.904079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:28.903540    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 01:53:28.904079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:28.903595    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 01:53:29.277138 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:29.276171    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 01:53:29.277138 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:29.276232    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 01:53:29.615121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:29.615091    4912 generic.go:356] "Generic (PLEG): container finished" podID="1554bc18-c67c-47b6-bc69-d12bf3042835" containerID="deb360c3d5a1e6171fcaa35c1b02848ff3784fccf3edbd5adade4dbb3f57fcd4" exitCode=0
Nov 02 01:53:29.615311 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:29.615179    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfrdp-6qbtd" event={"ID":"1554bc18-c67c-47b6-bc69-d12bf3042835","Type":"ContainerDied","Data":"deb360c3d5a1e6171fcaa35c1b02848ff3784fccf3edbd5adade4dbb3f57fcd4"}
Nov 02 01:53:29.675169 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:29.668576    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:29.903733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:29.903635    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 01:53:29.903733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:29.903691    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 01:53:30.270142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:30.269277    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 01:53:30.270142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:30.269343    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 01:53:30.909483 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:30.909449    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 01:53:30.910107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:30.909517    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 01:53:30.932346 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:30.932315    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:53:31.031219 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.031182    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1554bc18-c67c-47b6-bc69-d12bf3042835-kube-api-access-885r2\" (UniqueName: \"kubernetes.io/projected/1554bc18-c67c-47b6-bc69-d12bf3042835-kube-api-access-885r2\") pod \"1554bc18-c67c-47b6-bc69-d12bf3042835\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") "
Nov 02 01:53:31.031409 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.031243    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-service-ca-bundle\") pod \"1554bc18-c67c-47b6-bc69-d12bf3042835\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") "
Nov 02 01:53:31.031409 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.031319    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1554bc18-c67c-47b6-bc69-d12bf3042835-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1554bc18-c67c-47b6-bc69-d12bf3042835-archives-path\") pod \"1554bc18-c67c-47b6-bc69-d12bf3042835\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") "
Nov 02 01:53:31.031591 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.031572    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-trusted-ca-bundle\") pod \"1554bc18-c67c-47b6-bc69-d12bf3042835\" (UID: \"1554bc18-c67c-47b6-bc69-d12bf3042835\") "
Nov 02 01:53:31.032737 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.032699    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-service-ca-bundle" pod "1554bc18-c67c-47b6-bc69-d12bf3042835" (UID: "1554bc18-c67c-47b6-bc69-d12bf3042835"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:31.032864 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.032832    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554bc18-c67c-47b6-bc69-d12bf3042835-archives-path" pod "1554bc18-c67c-47b6-bc69-d12bf3042835" (UID: "1554bc18-c67c-47b6-bc69-d12bf3042835"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:53:31.032960 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.032521    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-trusted-ca-bundle" pod "1554bc18-c67c-47b6-bc69-d12bf3042835" (UID: "1554bc18-c67c-47b6-bc69-d12bf3042835"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:31.037619 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.037587    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1554bc18-c67c-47b6-bc69-d12bf3042835-kube-api-access-885r2" pod "1554bc18-c67c-47b6-bc69-d12bf3042835" (UID: "1554bc18-c67c-47b6-bc69-d12bf3042835"). InnerVolumeSpecName "kube-api-access-885r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:31.133783 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.133750    4912 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-trusted-ca-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:31.134021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.134001    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-885r2\" (UniqueName: \"kubernetes.io/projected/1554bc18-c67c-47b6-bc69-d12bf3042835-kube-api-access-885r2\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:31.134196 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.134181    4912 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1554bc18-c67c-47b6-bc69-d12bf3042835-service-ca-bundle\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:31.134291 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.134278    4912 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1554bc18-c67c-47b6-bc69-d12bf3042835-archives-path\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:31.284083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.283848    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:31.284485 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.284461    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:31.285522 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.285494    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 01:53:31.653962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.652118    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfrdp-6qbtd"
Nov 02 01:53:31.653962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.652239    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfrdp-6qbtd" event={"ID":"1554bc18-c67c-47b6-bc69-d12bf3042835","Type":"ContainerDied","Data":"ddaf5e43276af0bb43ca2813942ec1c9db33b5e288e160cfbc7c0ca42a097dce"}
Nov 02 01:53:31.653962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.652278    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddaf5e43276af0bb43ca2813942ec1c9db33b5e288e160cfbc7c0ca42a097dce"
Nov 02 01:53:31.903387 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.903355    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 01:53:31.903553 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:31.903415    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 01:53:32.024110 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.024024    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"]
Nov 02 01:53:32.024701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.024687    4912 cpu_manager.go:414] "containerMap: removing container" podUID="1554bc18-c67c-47b6-bc69-d12bf3042835" containerName="insights-gathering"
Nov 02 01:53:32.024740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.024703    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1554bc18-c67c-47b6-bc69-d12bf3042835" containerName="insights-gathering"
Nov 02 01:53:32.024817 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.024801    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="1554bc18-c67c-47b6-bc69-d12bf3042835" containerName="insights-gathering"
Nov 02 01:53:32.029324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.029302    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.081354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.081326    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:32.081528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.081474    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:32.084979 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.084953    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"]
Nov 02 01:53:32.165621 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.165585    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30561773-d579c\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.165802 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.165625    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhk9z\" (UniqueName: \"kubernetes.io/projected/99f17372-513e-4ca1-8ea6-1ed086c350b9-kube-api-access-fhk9z\") pod \"maas-api-revoke-sub-30561773-d579c\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.165850 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.165798    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30561773-d579c\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.267488 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.267417    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30561773-d579c\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.267488 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.267456    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhk9z\" (UniqueName: \"kubernetes.io/projected/99f17372-513e-4ca1-8ea6-1ed086c350b9-kube-api-access-fhk9z\") pod \"maas-api-revoke-sub-30561773-d579c\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.267780 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.267540    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30561773-d579c\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.268640 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.268608    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30561773-d579c\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.270128 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.270105    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30561773-d579c\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.328842 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.328755    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhk9z\" (UniqueName: \"kubernetes.io/projected/99f17372-513e-4ca1-8ea6-1ed086c350b9-kube-api-access-fhk9z\") pod \"maas-api-revoke-sub-30561773-d579c\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.344350 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.344317    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:32.728793 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:32.728763    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f17372_513e_4ca1_8ea6_1ed086c350b9.slice/crio-36e34699a9d0a2491a36c9da784bb3b093908b0f48f5ff3bbf08e59b4f40eccb WatchSource:0}: Error finding container 36e34699a9d0a2491a36c9da784bb3b093908b0f48f5ff3bbf08e59b4f40eccb: Status 404 returned error can't find the container with id 36e34699a9d0a2491a36c9da784bb3b093908b0f48f5ff3bbf08e59b4f40eccb
Nov 02 01:53:32.730463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.730431    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"]
Nov 02 01:53:32.846974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.846941    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:32.908194 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.908165    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:32.908646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.908627    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:32.909378 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:32.909361    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 01:53:33.024303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:33.024232    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:33.672156 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:33.672113    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c" event={"ID":"99f17372-513e-4ca1-8ea6-1ed086c350b9","Type":"ContainerStarted","Data":"361b3babb489dc8ca9be78f438c33a52a133aad12b7534cdfd0a34fc389e9a75"}
Nov 02 01:53:33.672349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:33.672185    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c" event={"ID":"99f17372-513e-4ca1-8ea6-1ed086c350b9","Type":"ContainerStarted","Data":"36e34699a9d0a2491a36c9da784bb3b093908b0f48f5ff3bbf08e59b4f40eccb"}
Nov 02 01:53:33.790250 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:33.790202    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c" podStartSLOduration=2.790188952 podStartE2EDuration="2.790188952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:33.786110551 +0000 UTC m=+1306.691575907" watchObservedRunningTime="2026-11-02 01:53:33.790188952 +0000 UTC m=+1306.695654307"
Nov 02 01:53:34.553602 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.553567    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:34.687517 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.687483    4912 generic.go:356] "Generic (PLEG): container finished" podID="99f17372-513e-4ca1-8ea6-1ed086c350b9" containerID="361b3babb489dc8ca9be78f438c33a52a133aad12b7534cdfd0a34fc389e9a75" exitCode=0
Nov 02 01:53:34.687728 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.687571    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c" event={"ID":"99f17372-513e-4ca1-8ea6-1ed086c350b9","Type":"ContainerDied","Data":"361b3babb489dc8ca9be78f438c33a52a133aad12b7534cdfd0a34fc389e9a75"}
Nov 02 01:53:34.733389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.733360    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"]
Nov 02 01:53:34.737472 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.737453    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.784112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.784069    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"]
Nov 02 01:53:34.788352 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.788326    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.799515 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.799491    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"]
Nov 02 01:53:34.811127 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.811012    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.811127 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.811099    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.811348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.811149    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.811348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.811218    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.811460 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.811349    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-674px\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-kube-api-access-674px\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.811460 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.811382    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.811460 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.811417    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.811636 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.811495    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.811636 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.811596    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.892708 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.892669    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"]
Nov 02 01:53:34.912666 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.912631    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.912864 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.912708    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.912864 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.912786    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.912864 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.912823    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-674px\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-kube-api-access-674px\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.912864 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.912851    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.913097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.912884    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.913097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.912939    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.913097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.912972    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbsn\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-kube-api-access-8zbsn\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.913097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.913004    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.913299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.913098    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.913299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.913123    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.913299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.913168    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.913299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.913196    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.913299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.913229    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.913650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.913631    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.913860 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.913837    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.913913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.913878    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.914215 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.914192    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.914453 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.914432    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.914937 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.914705    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.914937 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.914796    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:34.915247 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.915222    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.915350 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.915253    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.916793 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.916748    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.917133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.917088    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.975115 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.975087    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:34.975271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:34.975189    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-674px\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-kube-api-access-674px\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:35.016035 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016006    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016217 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016104    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016217 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016169    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016326 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016237    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016326 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016280    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbsn\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-kube-api-access-8zbsn\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016348    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016374    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016421    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016648 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016453    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016648 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016531    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016648 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016555    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016802 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016787    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.016975 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.016954    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.017511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.017483    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.018838 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.018805    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.019263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.019241    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.051644 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.051620    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:35.076956 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.076928    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.077481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.077457    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbsn\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-kube-api-access-8zbsn\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.104875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.104842    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:35.488398 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.488369    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"]
Nov 02 01:53:35.524037 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:35.523995    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d487f28_5c3b_4f6a_8259_b5f9c36fc88c.slice/crio-56b76f4496b1dc47b0cc0a7ceafb284a5a865b8b11f1d841c505f0f6e2e3d563 WatchSource:0}: Error finding container 56b76f4496b1dc47b0cc0a7ceafb284a5a865b8b11f1d841c505f0f6e2e3d563: Status 404 returned error can't find the container with id 56b76f4496b1dc47b0cc0a7ceafb284a5a865b8b11f1d841c505f0f6e2e3d563
Nov 02 01:53:35.526566 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.526534    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:35.526641 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.526625    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:35.526686 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.526654    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:35.540377 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:35.540353    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbfe2bb_a981_4339_8a0c_aa7d77994be0.slice/crio-9aabf4890eb624b30d1d18b525ef6afb324ce09760bf90d18811d46a583d380f WatchSource:0}: Error finding container 9aabf4890eb624b30d1d18b525ef6afb324ce09760bf90d18811d46a583d380f: Status 404 returned error can't find the container with id 9aabf4890eb624b30d1d18b525ef6afb324ce09760bf90d18811d46a583d380f
Nov 02 01:53:35.541796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.541774    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"]
Nov 02 01:53:35.543571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.543531    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:35.543660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.543598    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:35.543660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.543639    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:35.699824 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.699789    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" event={"ID":"7bbfe2bb-a981-4339-8a0c-aa7d77994be0","Type":"ContainerStarted","Data":"233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca"}
Nov 02 01:53:35.700301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.699845    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" event={"ID":"7bbfe2bb-a981-4339-8a0c-aa7d77994be0","Type":"ContainerStarted","Data":"9aabf4890eb624b30d1d18b525ef6afb324ce09760bf90d18811d46a583d380f"}
Nov 02 01:53:35.701615 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.701588    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" event={"ID":"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c","Type":"ContainerStarted","Data":"3f7af1341c149adf804d0a8c8ec66f3f1c7d1b712db3d473f9ecf2c89202a0c6"}
Nov 02 01:53:35.701759 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.701637    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" event={"ID":"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c","Type":"ContainerStarted","Data":"56b76f4496b1dc47b0cc0a7ceafb284a5a865b8b11f1d841c505f0f6e2e3d563"}
Nov 02 01:53:35.832631 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.832574    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podStartSLOduration=1.832555804 podStartE2EDuration="1.832555804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:35.824910444 +0000 UTC m=+1308.730375800" watchObservedRunningTime="2026-11-02 01:53:35.832555804 +0000 UTC m=+1308.738021161"
Nov 02 01:53:35.872733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.872707    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:35.976495 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:35.976424    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podStartSLOduration=1.976407849 podStartE2EDuration="1.976407849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:35.970598942 +0000 UTC m=+1308.876064312" watchObservedRunningTime="2026-11-02 01:53:35.976407849 +0000 UTC m=+1308.881873209"
Nov 02 01:53:36.038026 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.037933    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f17372-513e-4ca1-8ea6-1ed086c350b9-kube-api-access-fhk9z\" (UniqueName: \"kubernetes.io/projected/99f17372-513e-4ca1-8ea6-1ed086c350b9-kube-api-access-fhk9z\") pod \"99f17372-513e-4ca1-8ea6-1ed086c350b9\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") "
Nov 02 01:53:36.038026 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.038010    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-xks-ca\") pod \"99f17372-513e-4ca1-8ea6-1ed086c350b9\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") "
Nov 02 01:53:36.038279 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.038139    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-openshift-ca\") pod \"99f17372-513e-4ca1-8ea6-1ed086c350b9\" (UID: \"99f17372-513e-4ca1-8ea6-1ed086c350b9\") "
Nov 02 01:53:36.038483 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.038459    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-openshift-ca" pod "99f17372-513e-4ca1-8ea6-1ed086c350b9" (UID: "99f17372-513e-4ca1-8ea6-1ed086c350b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:36.038638 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.038620    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:36.040534 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.040506    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-xks-ca" pod "99f17372-513e-4ca1-8ea6-1ed086c350b9" (UID: "99f17372-513e-4ca1-8ea6-1ed086c350b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:36.040534 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.040527    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f17372-513e-4ca1-8ea6-1ed086c350b9-kube-api-access-fhk9z" pod "99f17372-513e-4ca1-8ea6-1ed086c350b9" (UID: "99f17372-513e-4ca1-8ea6-1ed086c350b9"). InnerVolumeSpecName "kube-api-access-fhk9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:36.052779 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.052741    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:36.054285 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.054254    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:53:36.054416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.054329    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 01:53:36.107083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.106148    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:36.117095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.113564    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:53:36.117095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.113619    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 01:53:36.148562 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.148525    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhk9z\" (UniqueName: \"kubernetes.io/projected/99f17372-513e-4ca1-8ea6-1ed086c350b9-kube-api-access-fhk9z\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:36.148562 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.148559    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f17372-513e-4ca1-8ea6-1ed086c350b9-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:36.717263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.717221    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c" event={"ID":"99f17372-513e-4ca1-8ea6-1ed086c350b9","Type":"ContainerDied","Data":"36e34699a9d0a2491a36c9da784bb3b093908b0f48f5ff3bbf08e59b4f40eccb"}
Nov 02 01:53:36.717763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.717285    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e34699a9d0a2491a36c9da784bb3b093908b0f48f5ff3bbf08e59b4f40eccb"
Nov 02 01:53:36.717763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:36.717317    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"
Nov 02 01:53:37.053523 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:37.053433    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:53:37.053786 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:37.053755    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 01:53:37.112083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:37.111158    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:53:37.112083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:37.111228    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 01:53:38.068081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.053410    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:53:38.068081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.053480    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 01:53:38.106081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.105633    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:53:38.106081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.105712    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 01:53:38.134713 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.134680    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"]
Nov 02 01:53:38.135339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.135323    4912 cpu_manager.go:414] "containerMap: removing container" podUID="99f17372-513e-4ca1-8ea6-1ed086c350b9" containerName="revoke-keys"
Nov 02 01:53:38.135397 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.135340    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f17372-513e-4ca1-8ea6-1ed086c350b9" containerName="revoke-keys"
Nov 02 01:53:38.135476 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.135467    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f17372-513e-4ca1-8ea6-1ed086c350b9" containerName="revoke-keys"
Nov 02 01:53:38.139306 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.139289    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.225996 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.225965    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7d7dab-52b3-4786-b338-0920e1f86129-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.226177 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.226008    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjdc\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-kube-api-access-sfjdc\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.226397 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.226372    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.226502 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.226488    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.228151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.226558    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.228151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.226637    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.228151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.226663    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.228151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.226706    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.228151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.226791    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.238958 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.238926    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"]
Nov 02 01:53:38.331777 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.331685    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.331777 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.331767    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.331805    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7d7dab-52b3-4786-b338-0920e1f86129-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.331832    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjdc\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-kube-api-access-sfjdc\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.331874    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.331934    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.331979    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.331999    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332324 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.332017    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332373 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.332355    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332582 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.332559    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.332814 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.332796    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.333024 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.333009    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.333578 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.333555    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7d7dab-52b3-4786-b338-0920e1f86129-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.335636 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.335604    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.336379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.336097    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.422656 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.422613    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjdc\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-kube-api-access-sfjdc\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.423614 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.423594    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:38.498081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:38.480310    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:39.021134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.020562    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"]
Nov 02 01:53:39.025712 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:39.025680    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7d7dab_52b3_4786_b338_0920e1f86129.slice/crio-f1ffac92410598f2d2fc46a367c1cabe2de32fb0a7cd2c6126c4dad75e708aa4 WatchSource:0}: Error finding container f1ffac92410598f2d2fc46a367c1cabe2de32fb0a7cd2c6126c4dad75e708aa4: Status 404 returned error can't find the container with id f1ffac92410598f2d2fc46a367c1cabe2de32fb0a7cd2c6126c4dad75e708aa4
Nov 02 01:53:39.029225 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.029189    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:39.029312 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.029294    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:39.029374 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.029356    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:53:39.074083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.073490    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:53:39.074083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.073547    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 01:53:39.115173 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.106969    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:53:39.115173 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.107099    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 01:53:39.447197 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.444653    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:39.793541 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.792270    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" event={"ID":"7a7d7dab-52b3-4786-b338-0920e1f86129","Type":"ContainerStarted","Data":"fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a"}
Nov 02 01:53:39.793541 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.792339    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" event={"ID":"7a7d7dab-52b3-4786-b338-0920e1f86129","Type":"ContainerStarted","Data":"f1ffac92410598f2d2fc46a367c1cabe2de32fb0a7cd2c6126c4dad75e708aa4"}
Nov 02 01:53:39.989851 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:39.988508    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podStartSLOduration=1.988488464 podStartE2EDuration="1.988488464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:39.974419002 +0000 UTC m=+1312.879884355" watchObservedRunningTime="2026-11-02 01:53:39.988488464 +0000 UTC m=+1312.893953820"
Nov 02 01:53:40.067148 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:40.066444    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:53:40.067148 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:40.066505    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 01:53:40.112082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:40.105223    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"]
Nov 02 01:53:40.112082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:40.105616    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:53:40.112082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:40.105664    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 01:53:40.118242 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:40.118195    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b89902-n6zg8"]
Nov 02 01:53:40.485080 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:40.484205    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:40.489079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:40.485997    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:53:40.489079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:40.486088    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 01:53:41.054281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:41.054003    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:53:41.054281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:41.054084    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 01:53:41.145225 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:41.130695    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:53:41.145225 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:41.130752    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 01:53:41.272083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:41.268941    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583d6858-964c-49b4-b8fa-6e3a59e16230" path="/var/lib/kubelet/pods/583d6858-964c-49b4-b8fa-6e3a59e16230/volumes"
Nov 02 01:53:41.499082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:41.498346    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:53:41.499082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:41.498406    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 01:53:42.053085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:42.052426    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:53:42.053085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:42.052485    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 01:53:42.111461 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:42.106010    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:53:42.111461 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:42.106110    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 01:53:42.140570 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:53:42.140532    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:53:42.482200 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:42.482164    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:53:42.483182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:42.482767    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 01:53:43.195085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:43.189444    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:43.195085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:43.190826    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:43.203083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:43.198279    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 01:53:43.262491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:43.262446    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:43.264352 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:43.264317    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:43.266313 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:43.266296    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 01:53:43.493128 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:43.492599    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:53:43.493128 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:43.492677    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 01:53:44.283083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:44.282481    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:44.481601 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:44.481566    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:53:44.481786 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:44.481634    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 01:53:45.048531 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:45.048500    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:45.482132 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:45.482094    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:53:45.482327 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:45.482153    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 01:53:46.481112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:46.481071    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:53:46.481669 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:46.481151    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 01:53:47.052641 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:47.052594    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"]
Nov 02 01:53:47.101917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:47.101885    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5b7976-9ddkb"]
Nov 02 01:53:47.227623 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:47.227593    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7716e24f-7066-448f-b110-a296f1faf33d" path="/var/lib/kubelet/pods/7716e24f-7066-448f-b110-a296f1faf33d/volumes"
Nov 02 01:53:47.486124 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:47.486100    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:47.486497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:47.486354    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:47.487154 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:47.487140    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 01:53:49.368560 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:49.368518    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:51.353038 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.353008    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"]
Nov 02 01:53:51.359326 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.359307    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.413026 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.412982    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqfd\" (UniqueName: \"kubernetes.io/projected/9fe0366c-d7e7-4593-84e9-73c34a91c10a-kube-api-access-xtqfd\") pod \"maas-api-revoke-sub-87f47c3a-5tq9b\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.413338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.413319    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f47c3a-5tq9b\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.413609 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.413589    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f47c3a-5tq9b\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.414558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.414532    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"]
Nov 02 01:53:51.457738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.457696    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:51.457950 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.457792    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:51.515103 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.515041    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqfd\" (UniqueName: \"kubernetes.io/projected/9fe0366c-d7e7-4593-84e9-73c34a91c10a-kube-api-access-xtqfd\") pod \"maas-api-revoke-sub-87f47c3a-5tq9b\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.515280 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.515150    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f47c3a-5tq9b\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.515280 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.515262    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f47c3a-5tq9b\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.515915 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.515896    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f47c3a-5tq9b\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.517642 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.517613    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f47c3a-5tq9b\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.575897 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.575869    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqfd\" (UniqueName: \"kubernetes.io/projected/9fe0366c-d7e7-4593-84e9-73c34a91c10a-kube-api-access-xtqfd\") pod \"maas-api-revoke-sub-87f47c3a-5tq9b\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:51.676924 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:51.676894    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:52.067220 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:52.067193    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe0366c_d7e7_4593_84e9_73c34a91c10a.slice/crio-885dc05285d2035852c1a7865a848e6a901c8c691a397fb8b0b6c92276352b22 WatchSource:0}: Error finding container 885dc05285d2035852c1a7865a848e6a901c8c691a397fb8b0b6c92276352b22: Status 404 returned error can't find the container with id 885dc05285d2035852c1a7865a848e6a901c8c691a397fb8b0b6c92276352b22
Nov 02 01:53:52.069626 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:52.069596    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"]
Nov 02 01:53:53.059181 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:53.059144    4912 generic.go:356] "Generic (PLEG): container finished" podID="9fe0366c-d7e7-4593-84e9-73c34a91c10a" containerID="2c28938e0e3b02da3f5163df48f04b9927d048a714eb45aa2acdde32ac59a7b2" exitCode=0
Nov 02 01:53:53.059608 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:53.059215    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b" event={"ID":"9fe0366c-d7e7-4593-84e9-73c34a91c10a","Type":"ContainerDied","Data":"2c28938e0e3b02da3f5163df48f04b9927d048a714eb45aa2acdde32ac59a7b2"}
Nov 02 01:53:53.059608 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:53.059280    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b" event={"ID":"9fe0366c-d7e7-4593-84e9-73c34a91c10a","Type":"ContainerStarted","Data":"885dc05285d2035852c1a7865a848e6a901c8c691a397fb8b0b6c92276352b22"}
Nov 02 01:53:54.073514 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.073477    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:54.271229 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.271202    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:54.356579 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.356500    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe0366c-d7e7-4593-84e9-73c34a91c10a-kube-api-access-xtqfd\" (UniqueName: \"kubernetes.io/projected/9fe0366c-d7e7-4593-84e9-73c34a91c10a-kube-api-access-xtqfd\") pod \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") "
Nov 02 01:53:54.356719 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.356689    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-xks-ca\") pod \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") "
Nov 02 01:53:54.356769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.356724    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-openshift-ca\") pod \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\" (UID: \"9fe0366c-d7e7-4593-84e9-73c34a91c10a\") "
Nov 02 01:53:54.357369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.357169    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-openshift-ca" pod "9fe0366c-d7e7-4593-84e9-73c34a91c10a" (UID: "9fe0366c-d7e7-4593-84e9-73c34a91c10a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:54.357369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.357338    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:54.358836 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.358812    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe0366c-d7e7-4593-84e9-73c34a91c10a-kube-api-access-xtqfd" pod "9fe0366c-d7e7-4593-84e9-73c34a91c10a" (UID: "9fe0366c-d7e7-4593-84e9-73c34a91c10a"). InnerVolumeSpecName "kube-api-access-xtqfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:54.358836 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.358822    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-xks-ca" pod "9fe0366c-d7e7-4593-84e9-73c34a91c10a" (UID: "9fe0366c-d7e7-4593-84e9-73c34a91c10a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:54.458768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.458735    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtqfd\" (UniqueName: \"kubernetes.io/projected/9fe0366c-d7e7-4593-84e9-73c34a91c10a-kube-api-access-xtqfd\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:54.458768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:54.458767    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe0366c-d7e7-4593-84e9-73c34a91c10a-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:55.078422 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:55.078380    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b" event={"ID":"9fe0366c-d7e7-4593-84e9-73c34a91c10a","Type":"ContainerDied","Data":"885dc05285d2035852c1a7865a848e6a901c8c691a397fb8b0b6c92276352b22"}
Nov 02 01:53:55.079095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:55.078482    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="885dc05285d2035852c1a7865a848e6a901c8c691a397fb8b0b6c92276352b22"
Nov 02 01:53:55.079095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:55.078559    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"
Nov 02 01:53:56.959278 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:56.959247    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"]
Nov 02 01:53:56.959919 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:56.959904    4912 cpu_manager.go:414] "containerMap: removing container" podUID="9fe0366c-d7e7-4593-84e9-73c34a91c10a" containerName="revoke-keys"
Nov 02 01:53:56.959961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:56.959921    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe0366c-d7e7-4593-84e9-73c34a91c10a" containerName="revoke-keys"
Nov 02 01:53:56.960025 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:56.960016    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe0366c-d7e7-4593-84e9-73c34a91c10a" containerName="revoke-keys"
Nov 02 01:53:56.963856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:56.963835    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:56.992809 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:56.992787    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b488eb32-5rqht\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:56.992924 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:56.992835    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b488eb32-5rqht\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:56.993110 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:56.993092    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxwj\" (UniqueName: \"kubernetes.io/projected/3b91b946-671e-48d8-bd8d-f309653460fe-kube-api-access-6cxwj\") pod \"maas-api-revoke-sub-b488eb32-5rqht\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:57.022849 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.022825    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"]
Nov 02 01:53:57.065314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.065289    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:57.065468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.065405    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:57.094947 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.094901    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxwj\" (UniqueName: \"kubernetes.io/projected/3b91b946-671e-48d8-bd8d-f309653460fe-kube-api-access-6cxwj\") pod \"maas-api-revoke-sub-b488eb32-5rqht\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:57.095157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.095091    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b488eb32-5rqht\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:57.095157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.095151    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b488eb32-5rqht\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:57.095911 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.095889    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b488eb32-5rqht\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:57.097673 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.097651    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b488eb32-5rqht\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:57.156210 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.156178    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxwj\" (UniqueName: \"kubernetes.io/projected/3b91b946-671e-48d8-bd8d-f309653460fe-kube-api-access-6cxwj\") pod \"maas-api-revoke-sub-b488eb32-5rqht\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:57.277878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.277789    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:53:57.664098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:57.664069    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"]
Nov 02 01:53:57.664798 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:53:57.664772    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b91b946_671e_48d8_bd8d_f309653460fe.slice/crio-b156912842c2fe4eb7e2f14ef3dfd917d0595eacfe718bf49e111bbf16477345 WatchSource:0}: Error finding container b156912842c2fe4eb7e2f14ef3dfd917d0595eacfe718bf49e111bbf16477345: Status 404 returned error can't find the container with id b156912842c2fe4eb7e2f14ef3dfd917d0595eacfe718bf49e111bbf16477345
Nov 02 01:53:58.105322 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:58.105261    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht" event={"ID":"3b91b946-671e-48d8-bd8d-f309653460fe","Type":"ContainerStarted","Data":"64f17248b145f4c7e9385f2dcdcfc28889ef9817f12ef4a3de810854c9861987"}
Nov 02 01:53:58.105720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:58.105370    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht" event={"ID":"3b91b946-671e-48d8-bd8d-f309653460fe","Type":"ContainerStarted","Data":"b156912842c2fe4eb7e2f14ef3dfd917d0595eacfe718bf49e111bbf16477345"}
Nov 02 01:53:58.223441 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:58.223351    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht" podStartSLOduration=2.22333828 podStartE2EDuration="2.22333828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:58.221743847 +0000 UTC m=+1331.127209204" watchObservedRunningTime="2026-11-02 01:53:58.22333828 +0000 UTC m=+1331.128803635"
Nov 02 01:53:58.862160 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:58.862051    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:53:59.115169 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:59.115140    4912 generic.go:356] "Generic (PLEG): container finished" podID="3b91b946-671e-48d8-bd8d-f309653460fe" containerID="64f17248b145f4c7e9385f2dcdcfc28889ef9817f12ef4a3de810854c9861987" exitCode=0
Nov 02 01:53:59.115598 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:53:59.115231    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht" event={"ID":"3b91b946-671e-48d8-bd8d-f309653460fe","Type":"ContainerDied","Data":"64f17248b145f4c7e9385f2dcdcfc28889ef9817f12ef4a3de810854c9861987"}
Nov 02 01:54:00.303835 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.303812    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:54:00.340334 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.340303    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-openshift-ca\") pod \"3b91b946-671e-48d8-bd8d-f309653460fe\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") "
Nov 02 01:54:00.340493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.340407    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b91b946-671e-48d8-bd8d-f309653460fe-kube-api-access-6cxwj\" (UniqueName: \"kubernetes.io/projected/3b91b946-671e-48d8-bd8d-f309653460fe-kube-api-access-6cxwj\") pod \"3b91b946-671e-48d8-bd8d-f309653460fe\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") "
Nov 02 01:54:00.340562 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.340494    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-xks-ca\") pod \"3b91b946-671e-48d8-bd8d-f309653460fe\" (UID: \"3b91b946-671e-48d8-bd8d-f309653460fe\") "
Nov 02 01:54:00.340744 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.340708    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-openshift-ca" pod "3b91b946-671e-48d8-bd8d-f309653460fe" (UID: "3b91b946-671e-48d8-bd8d-f309653460fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:00.341086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.341032    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:00.342745 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.342720    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b91b946-671e-48d8-bd8d-f309653460fe-kube-api-access-6cxwj" pod "3b91b946-671e-48d8-bd8d-f309653460fe" (UID: "3b91b946-671e-48d8-bd8d-f309653460fe"). InnerVolumeSpecName "kube-api-access-6cxwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:00.343079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.343032    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-xks-ca" pod "3b91b946-671e-48d8-bd8d-f309653460fe" (UID: "3b91b946-671e-48d8-bd8d-f309653460fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:00.442403 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.442325    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxwj\" (UniqueName: \"kubernetes.io/projected/3b91b946-671e-48d8-bd8d-f309653460fe-kube-api-access-6cxwj\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:00.442403 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:00.442355    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b91b946-671e-48d8-bd8d-f309653460fe-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:01.056629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.056591    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"]
Nov 02 01:54:01.061435 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.061410    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ff421-r4hkq"]
Nov 02 01:54:01.128595 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.128563    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"]
Nov 02 01:54:01.129259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.129244    4912 cpu_manager.go:414] "containerMap: removing container" podUID="3b91b946-671e-48d8-bd8d-f309653460fe" containerName="revoke-keys"
Nov 02 01:54:01.129317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.129261    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b91b946-671e-48d8-bd8d-f309653460fe" containerName="revoke-keys"
Nov 02 01:54:01.129387 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.129377    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b91b946-671e-48d8-bd8d-f309653460fe" containerName="revoke-keys"
Nov 02 01:54:01.133715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.133691    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.133844 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.133750    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"
Nov 02 01:54:01.133844 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.133772    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht" event={"ID":"3b91b946-671e-48d8-bd8d-f309653460fe","Type":"ContainerDied","Data":"b156912842c2fe4eb7e2f14ef3dfd917d0595eacfe718bf49e111bbf16477345"}
Nov 02 01:54:01.133844 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.133808    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b156912842c2fe4eb7e2f14ef3dfd917d0595eacfe718bf49e111bbf16477345"
Nov 02 01:54:01.212842 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.212803    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"]
Nov 02 01:54:01.231014 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.230985    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8679b55-4cc4-4740-8d0e-c71bbb616175" path="/var/lib/kubelet/pods/a8679b55-4cc4-4740-8d0e-c71bbb616175/volumes"
Nov 02 01:54:01.255164 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.255133    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-188623e0-g5glw\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.255349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.255254    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-188623e0-g5glw\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.255418 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.255395    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9tsf\" (UniqueName: \"kubernetes.io/projected/4bae8290-d9d6-4281-8d86-518aa41e11e1-kube-api-access-n9tsf\") pod \"maas-api-revoke-sub-188623e0-g5glw\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.356746 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.356652    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-188623e0-g5glw\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.357305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.356760    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-188623e0-g5glw\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.357305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.357023    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9tsf\" (UniqueName: \"kubernetes.io/projected/4bae8290-d9d6-4281-8d86-518aa41e11e1-kube-api-access-n9tsf\") pod \"maas-api-revoke-sub-188623e0-g5glw\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.357596 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.357571    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-188623e0-g5glw\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.359661 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.359640    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-188623e0-g5glw\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.418050 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.418018    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9tsf\" (UniqueName: \"kubernetes.io/projected/4bae8290-d9d6-4281-8d86-518aa41e11e1-kube-api-access-n9tsf\") pod \"maas-api-revoke-sub-188623e0-g5glw\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.453169 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.453141    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:01.841544 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:01.841513    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bae8290_d9d6_4281_8d86_518aa41e11e1.slice/crio-b958ed2fe13868883e9915deb37f27532cdebc71b2d98bfe4ab595ebf21bf6a5 WatchSource:0}: Error finding container b958ed2fe13868883e9915deb37f27532cdebc71b2d98bfe4ab595ebf21bf6a5: Status 404 returned error can't find the container with id b958ed2fe13868883e9915deb37f27532cdebc71b2d98bfe4ab595ebf21bf6a5
Nov 02 01:54:01.842717 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:01.842683    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"]
Nov 02 01:54:02.143878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:02.143847    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw" event={"ID":"4bae8290-d9d6-4281-8d86-518aa41e11e1","Type":"ContainerStarted","Data":"b958ed2fe13868883e9915deb37f27532cdebc71b2d98bfe4ab595ebf21bf6a5"}
Nov 02 01:54:03.049679 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:03.049647    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"]
Nov 02 01:54:03.099571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:03.099487    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564a41a1-8wsh5"]
Nov 02 01:54:03.154872 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:03.154842    4912 generic.go:356] "Generic (PLEG): container finished" podID="4bae8290-d9d6-4281-8d86-518aa41e11e1" containerID="4b9e40f9913f2ae14dddd5cec93e4aa535e5007f350f14cfb4a99c8c435e3f0f" exitCode=0
Nov 02 01:54:03.155116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:03.154932    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw" event={"ID":"4bae8290-d9d6-4281-8d86-518aa41e11e1","Type":"ContainerDied","Data":"4b9e40f9913f2ae14dddd5cec93e4aa535e5007f350f14cfb4a99c8c435e3f0f"}
Nov 02 01:54:03.236545 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:03.236515    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8788c8f3-2c96-4503-8897-8fb7ea516c9e" path="/var/lib/kubelet/pods/8788c8f3-2c96-4503-8897-8fb7ea516c9e/volumes"
Nov 02 01:54:03.706389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:03.706344    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:04.314368 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.314343    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:04.398520 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.398484    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-xks-ca\") pod \"4bae8290-d9d6-4281-8d86-518aa41e11e1\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") "
Nov 02 01:54:04.398687 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.398636    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bae8290-d9d6-4281-8d86-518aa41e11e1-kube-api-access-n9tsf\" (UniqueName: \"kubernetes.io/projected/4bae8290-d9d6-4281-8d86-518aa41e11e1-kube-api-access-n9tsf\") pod \"4bae8290-d9d6-4281-8d86-518aa41e11e1\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") "
Nov 02 01:54:04.398687 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.398683    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-openshift-ca\") pod \"4bae8290-d9d6-4281-8d86-518aa41e11e1\" (UID: \"4bae8290-d9d6-4281-8d86-518aa41e11e1\") "
Nov 02 01:54:04.399089 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.399028    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-openshift-ca" pod "4bae8290-d9d6-4281-8d86-518aa41e11e1" (UID: "4bae8290-d9d6-4281-8d86-518aa41e11e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:04.400828 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.400809    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-xks-ca" pod "4bae8290-d9d6-4281-8d86-518aa41e11e1" (UID: "4bae8290-d9d6-4281-8d86-518aa41e11e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:04.400939 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.400920    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bae8290-d9d6-4281-8d86-518aa41e11e1-kube-api-access-n9tsf" pod "4bae8290-d9d6-4281-8d86-518aa41e11e1" (UID: "4bae8290-d9d6-4281-8d86-518aa41e11e1"). InnerVolumeSpecName "kube-api-access-n9tsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:04.500699 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.500614    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9tsf\" (UniqueName: \"kubernetes.io/projected/4bae8290-d9d6-4281-8d86-518aa41e11e1-kube-api-access-n9tsf\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:04.500699 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.500645    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:04.500699 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:04.500658    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bae8290-d9d6-4281-8d86-518aa41e11e1-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:05.174044 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:05.174013    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"
Nov 02 01:54:05.174247 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:05.174040    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw" event={"ID":"4bae8290-d9d6-4281-8d86-518aa41e11e1","Type":"ContainerDied","Data":"b958ed2fe13868883e9915deb37f27532cdebc71b2d98bfe4ab595ebf21bf6a5"}
Nov 02 01:54:05.174247 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:05.174120    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b958ed2fe13868883e9915deb37f27532cdebc71b2d98bfe4ab595ebf21bf6a5"
Nov 02 01:54:06.049105 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:06.049072    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"]
Nov 02 01:54:06.098799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:06.098769    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8899d465-hxvpj"]
Nov 02 01:54:07.235311 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:07.235261    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc" path="/var/lib/kubelet/pods/c49fb19e-9e56-4568-8e6f-0b3c0f3fa1cc/volumes"
Nov 02 01:54:08.543962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:08.543926    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:09.523219 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.523186    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"]
Nov 02 01:54:09.523842 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.523828    4912 cpu_manager.go:414] "containerMap: removing container" podUID="4bae8290-d9d6-4281-8d86-518aa41e11e1" containerName="revoke-keys"
Nov 02 01:54:09.523897 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.523843    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bae8290-d9d6-4281-8d86-518aa41e11e1" containerName="revoke-keys"
Nov 02 01:54:09.523960 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.523951    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bae8290-d9d6-4281-8d86-518aa41e11e1" containerName="revoke-keys"
Nov 02 01:54:09.528788 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.528770    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.636601 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.636572    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"]
Nov 02 01:54:09.670175 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.670135    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8sq\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-kube-api-access-sx8sq\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.670336 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.670191    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.670336 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.670275    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.670476 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.670336    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.670476 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.670401    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.670592 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.670475    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.670592 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.670500    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.670696 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.670646    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.670696 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.670673    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.771818 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.771789    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.771818 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.771826    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.771920    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8sq\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-kube-api-access-sx8sq\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.771968    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.772003    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.772105    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.772143    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.772172    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.772210    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772354 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.772346    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772817 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.772606    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.772950 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.772934    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.773170 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.773009    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.773170 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.773005    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.774731 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.774681    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.775415 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.775398    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.833658 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.833631    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.833838 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.833818    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8sq\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-kube-api-access-sx8sq\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:09.843004 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:09.842979    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:10.278251 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.278085    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"]
Nov 02 01:54:10.281954 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:10.281921    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e945abf_cbdf_44ea_ad4f_59aaaa82b264.slice/crio-075f7a30c74730ed1a46d11801b8b37eb619d167f24d7ea2fa96843c8f34a83c WatchSource:0}: Error finding container 075f7a30c74730ed1a46d11801b8b37eb619d167f24d7ea2fa96843c8f34a83c: Status 404 returned error can't find the container with id 075f7a30c74730ed1a46d11801b8b37eb619d167f24d7ea2fa96843c8f34a83c
Nov 02 01:54:10.284783 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.284747    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:54:10.284856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.284839    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:54:10.284895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.284880    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:54:10.877082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.876914    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"]
Nov 02 01:54:10.888050 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.888014    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:10.920884 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.920508    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"]
Nov 02 01:54:10.967402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.967315    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:10.967402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.967370    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:10.992600 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.992555    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84de8c02-bbxqc\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:10.992867 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.992835    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29t8m\" (UniqueName: \"kubernetes.io/projected/53f88ec0-3de0-459f-bb7a-744218cf3c4d-kube-api-access-29t8m\") pod \"maas-api-revoke-sub-84de8c02-bbxqc\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:10.992965 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:10.992899    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84de8c02-bbxqc\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:11.094605 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.094573    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29t8m\" (UniqueName: \"kubernetes.io/projected/53f88ec0-3de0-459f-bb7a-744218cf3c4d-kube-api-access-29t8m\") pod \"maas-api-revoke-sub-84de8c02-bbxqc\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:11.094784 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.094619    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84de8c02-bbxqc\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:11.094784 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.094717    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84de8c02-bbxqc\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:11.095490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.095404    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84de8c02-bbxqc\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:11.097235 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.097205    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84de8c02-bbxqc\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:11.182859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.182775    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29t8m\" (UniqueName: \"kubernetes.io/projected/53f88ec0-3de0-459f-bb7a-744218cf3c4d-kube-api-access-29t8m\") pod \"maas-api-revoke-sub-84de8c02-bbxqc\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:11.207242 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.207207    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:11.228683 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.228649    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" event={"ID":"8e945abf-cbdf-44ea-ad4f-59aaaa82b264","Type":"ContainerStarted","Data":"32f6ec00a31e69b8100fef64ac75517200e84d84b21432350081410cc5e05efd"}
Nov 02 01:54:11.228791 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.228710    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" event={"ID":"8e945abf-cbdf-44ea-ad4f-59aaaa82b264","Type":"ContainerStarted","Data":"075f7a30c74730ed1a46d11801b8b37eb619d167f24d7ea2fa96843c8f34a83c"}
Nov 02 01:54:11.386459 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.386402    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" podStartSLOduration=2.386385281 podStartE2EDuration="2.386385281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:11.380995218 +0000 UTC m=+1344.286460576" watchObservedRunningTime="2026-11-02 01:54:11.386385281 +0000 UTC m=+1344.291850636"
Nov 02 01:54:11.621842 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.621815    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"]
Nov 02 01:54:11.624097 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:11.622383    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f88ec0_3de0_459f_bb7a_744218cf3c4d.slice/crio-6cd314b6fdd1635ab903e23c97ba42ccc827e7f41a545f889ae1199ecbf62a1b WatchSource:0}: Error finding container 6cd314b6fdd1635ab903e23c97ba42ccc827e7f41a545f889ae1199ecbf62a1b: Status 404 returned error can't find the container with id 6cd314b6fdd1635ab903e23c97ba42ccc827e7f41a545f889ae1199ecbf62a1b
Nov 02 01:54:11.844145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.844101    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:11.845811 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.845786    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:54:11.845935 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:11.845833    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:54:12.097136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:12.096547    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:12.147297 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:54:12.147219    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:54:12.238015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:12.237972    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc" event={"ID":"53f88ec0-3de0-459f-bb7a-744218cf3c4d","Type":"ContainerStarted","Data":"823249985bd088549abae319e572505735ac0b3d0588a95384b2443fbe39ba6c"}
Nov 02 01:54:12.238203 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:12.238031    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc" event={"ID":"53f88ec0-3de0-459f-bb7a-744218cf3c4d","Type":"ContainerStarted","Data":"6cd314b6fdd1635ab903e23c97ba42ccc827e7f41a545f889ae1199ecbf62a1b"}
Nov 02 01:54:12.282908 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:12.282868    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:12.385884 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:12.385825    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc" podStartSLOduration=2.385810653 podStartE2EDuration="2.385810653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:12.363598922 +0000 UTC m=+1345.269064278" watchObservedRunningTime="2026-11-02 01:54:12.385810653 +0000 UTC m=+1345.291276008"
Nov 02 01:54:12.843641 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:12.843615    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:54:12.843754 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:12.843677    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:54:13.247290 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:13.247258    4912 generic.go:356] "Generic (PLEG): container finished" podID="53f88ec0-3de0-459f-bb7a-744218cf3c4d" containerID="823249985bd088549abae319e572505735ac0b3d0588a95384b2443fbe39ba6c" exitCode=0
Nov 02 01:54:13.247835 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:13.247392    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc" event={"ID":"53f88ec0-3de0-459f-bb7a-744218cf3c4d","Type":"ContainerDied","Data":"823249985bd088549abae319e572505735ac0b3d0588a95384b2443fbe39ba6c"}
Nov 02 01:54:13.535412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:13.535331    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:13.843479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:13.843397    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:54:13.843479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:13.843450    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:54:14.503490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.503465    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:14.544542 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.544476    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-xks-ca\") pod \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") "
Nov 02 01:54:14.544744 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.544728    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f88ec0-3de0-459f-bb7a-744218cf3c4d-kube-api-access-29t8m\" (UniqueName: \"kubernetes.io/projected/53f88ec0-3de0-459f-bb7a-744218cf3c4d-kube-api-access-29t8m\") pod \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") "
Nov 02 01:54:14.544809 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.544780    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-openshift-ca\") pod \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\" (UID: \"53f88ec0-3de0-459f-bb7a-744218cf3c4d\") "
Nov 02 01:54:14.545165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.545123    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-openshift-ca" pod "53f88ec0-3de0-459f-bb7a-744218cf3c4d" (UID: "53f88ec0-3de0-459f-bb7a-744218cf3c4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:14.545359 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.545341    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:14.546696 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.546665    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-xks-ca" pod "53f88ec0-3de0-459f-bb7a-744218cf3c4d" (UID: "53f88ec0-3de0-459f-bb7a-744218cf3c4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:14.546771 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.546733    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f88ec0-3de0-459f-bb7a-744218cf3c4d-kube-api-access-29t8m" pod "53f88ec0-3de0-459f-bb7a-744218cf3c4d" (UID: "53f88ec0-3de0-459f-bb7a-744218cf3c4d"). InnerVolumeSpecName "kube-api-access-29t8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:14.646293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.646262    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29t8m\" (UniqueName: \"kubernetes.io/projected/53f88ec0-3de0-459f-bb7a-744218cf3c4d-kube-api-access-29t8m\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:14.646293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.646292    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f88ec0-3de0-459f-bb7a-744218cf3c4d-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:14.843629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.843553    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:54:14.843629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:14.843616    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:54:15.085080 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:15.084492    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"]
Nov 02 01:54:15.092000 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:15.091958    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5a7569-zzqdq"]
Nov 02 01:54:15.227468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:15.227437    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3a4a0e-4569-4a21-9d23-d60bd4053668" path="/var/lib/kubelet/pods/8c3a4a0e-4569-4a21-9d23-d60bd4053668/volumes"
Nov 02 01:54:15.290310 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:15.290282    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"
Nov 02 01:54:15.290938 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:15.290280    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc" event={"ID":"53f88ec0-3de0-459f-bb7a-744218cf3c4d","Type":"ContainerDied","Data":"6cd314b6fdd1635ab903e23c97ba42ccc827e7f41a545f889ae1199ecbf62a1b"}
Nov 02 01:54:15.291024 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:15.290964    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cd314b6fdd1635ab903e23c97ba42ccc827e7f41a545f889ae1199ecbf62a1b"
Nov 02 01:54:15.843350 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:15.843313    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:54:15.843759 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:15.843364    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:54:16.844025 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:16.843993    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:54:16.844468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:16.844092    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:54:17.050310 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:17.050259    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"]
Nov 02 01:54:17.103031 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:17.102951    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719efb0a-jrddr"]
Nov 02 01:54:17.233802 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:17.233758    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e5e5e0-833c-41a9-8113-87321325edd2" path="/var/lib/kubelet/pods/d1e5e5e0-833c-41a9-8113-87321325edd2/volumes"
Nov 02 01:54:17.847848 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:17.847820    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:17.848334 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:17.848311    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:17.849232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:17.849213    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 01:54:18.492488 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:18.492449    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:22.135084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:22.126149    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"]
Nov 02 01:54:22.135084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:22.130928    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ced7c18-vkpvd"]
Nov 02 01:54:22.415261 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:22.415156    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.612204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:22.612151    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.731133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:22.724159    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.754087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:22.753164    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.809085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:22.808170    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.836083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:22.835142    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 01:54:22.845191 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:22.845143    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.039314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.039125    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 02 01:54:23.066205 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.066152    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.232709 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.232663    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.298823 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.297190    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.423089 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.388222    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.512414 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.511196    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.641245 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.641202    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.750782 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.750739    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 01:54:23.777246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.774503    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.854131 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:23.824448    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:24.040007 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.039900    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:24.046647 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.046603    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690cbd4f-02ac-4d53-9bd8-0eb98a249f81" path="/var/lib/kubelet/pods/690cbd4f-02ac-4d53-9bd8-0eb98a249f81/volumes"
Nov 02 01:54:24.047760 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.047742    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:54:24.047994 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.047962    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:54:24.048701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.048674    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:54:24.048803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.048720    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:54:24.048803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.048753    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:54:24.048803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.048786    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:54:24.048970 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.048813    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:54:24.070095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.069972    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:54:24.132051 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.130666    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:54:24.159979 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.159949    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:54:24.177551 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.177336    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:54:24.208152 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.208120    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:54:24.822263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.822231    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"]
Nov 02 01:54:24.823282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.823258    4912 cpu_manager.go:414] "containerMap: removing container" podUID="53f88ec0-3de0-459f-bb7a-744218cf3c4d" containerName="revoke-keys"
Nov 02 01:54:24.824035 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.823419    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f88ec0-3de0-459f-bb7a-744218cf3c4d" containerName="revoke-keys"
Nov 02 01:54:24.824035 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.823696    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f88ec0-3de0-459f-bb7a-744218cf3c4d" containerName="revoke-keys"
Nov 02 01:54:24.828294 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.828256    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:24.880612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.880559    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:24.882654 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.882632    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"]
Nov 02 01:54:24.883102 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.883078    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:24.937558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.937524    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsld6\" (UniqueName: \"kubernetes.io/projected/33581af1-9dd2-4f8a-9bfa-49537e9c652c-kube-api-access-rsld6\") pod \"maas-api-revoke-sub-ff285d9d-xmrmr\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:24.937732 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.937617    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff285d9d-xmrmr\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:24.937823 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:24.937787    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff285d9d-xmrmr\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:25.039074 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.039020    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsld6\" (UniqueName: \"kubernetes.io/projected/33581af1-9dd2-4f8a-9bfa-49537e9c652c-kube-api-access-rsld6\") pod \"maas-api-revoke-sub-ff285d9d-xmrmr\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:25.039275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.039083    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff285d9d-xmrmr\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:25.039275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.039152    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff285d9d-xmrmr\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:25.039913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.039889    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff285d9d-xmrmr\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:25.041742 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.041713    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff285d9d-xmrmr\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:25.099989 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.099908    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsld6\" (UniqueName: \"kubernetes.io/projected/33581af1-9dd2-4f8a-9bfa-49537e9c652c-kube-api-access-rsld6\") pod \"maas-api-revoke-sub-ff285d9d-xmrmr\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:25.152766 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.152733    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:25.422370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.422341    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"]
Nov 02 01:54:25.427908 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.427891    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.485908 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.485859    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"]
Nov 02 01:54:25.555319 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.555289    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7k2l\" (UniqueName: \"kubernetes.io/projected/a1c8dc74-fef9-454e-90c8-b844df4403a3-kube-api-access-n7k2l\") pod \"maas-api-revoke-sub-b2293eee-ndjtb\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.555477 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.555423    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2293eee-ndjtb\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.555477 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.555460    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2293eee-ndjtb\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.594562 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:25.594346    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33581af1_9dd2_4f8a_9bfa_49537e9c652c.slice/crio-945d26ad5fc5fb276919008c87d1eb9b9882d97463d8c192c7877a38162f193a WatchSource:0}: Error finding container 945d26ad5fc5fb276919008c87d1eb9b9882d97463d8c192c7877a38162f193a: Status 404 returned error can't find the container with id 945d26ad5fc5fb276919008c87d1eb9b9882d97463d8c192c7877a38162f193a
Nov 02 01:54:25.596633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.596609    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"]
Nov 02 01:54:25.656781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.656739    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7k2l\" (UniqueName: \"kubernetes.io/projected/a1c8dc74-fef9-454e-90c8-b844df4403a3-kube-api-access-n7k2l\") pod \"maas-api-revoke-sub-b2293eee-ndjtb\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.656942 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.656879    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2293eee-ndjtb\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.657005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.656924    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2293eee-ndjtb\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.657660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.657640    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2293eee-ndjtb\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.659901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.659880    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2293eee-ndjtb\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.717690 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.717613    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7k2l\" (UniqueName: \"kubernetes.io/projected/a1c8dc74-fef9-454e-90c8-b844df4403a3-kube-api-access-n7k2l\") pod \"maas-api-revoke-sub-b2293eee-ndjtb\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.743380 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.743349    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:25.790316 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:25.790284    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:26.049076 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:26.049023    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"]
Nov 02 01:54:26.055511 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:26.055483    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c707249b-ncm86"]
Nov 02 01:54:26.205023 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:26.204982    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr" event={"ID":"33581af1-9dd2-4f8a-9bfa-49537e9c652c","Type":"ContainerStarted","Data":"6e322616bfdacb601850bafbcc5efafb9f28441701cfc12a6d0330c02ba89c00"}
Nov 02 01:54:26.205214 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:26.205043    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr" event={"ID":"33581af1-9dd2-4f8a-9bfa-49537e9c652c","Type":"ContainerStarted","Data":"945d26ad5fc5fb276919008c87d1eb9b9882d97463d8c192c7877a38162f193a"}
Nov 02 01:54:26.277822 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:26.277791    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c8dc74_fef9_454e_90c8_b844df4403a3.slice/crio-d0ceb35fa8078d2360b75a1b891cde5fb17c6f25d1f06f50d95e566fbc87a384 WatchSource:0}: Error finding container d0ceb35fa8078d2360b75a1b891cde5fb17c6f25d1f06f50d95e566fbc87a384: Status 404 returned error can't find the container with id d0ceb35fa8078d2360b75a1b891cde5fb17c6f25d1f06f50d95e566fbc87a384
Nov 02 01:54:26.278994 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:26.278970    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"]
Nov 02 01:54:26.320339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:26.320295    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr" podStartSLOduration=2.32028052 podStartE2EDuration="2.32028052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:26.319750475 +0000 UTC m=+1359.225215838" watchObservedRunningTime="2026-11-02 01:54:26.32028052 +0000 UTC m=+1359.225745878"
Nov 02 01:54:27.216743 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:27.216712    4912 generic.go:356] "Generic (PLEG): container finished" podID="33581af1-9dd2-4f8a-9bfa-49537e9c652c" containerID="6e322616bfdacb601850bafbcc5efafb9f28441701cfc12a6d0330c02ba89c00" exitCode=0
Nov 02 01:54:27.217202 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:27.216931    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr" event={"ID":"33581af1-9dd2-4f8a-9bfa-49537e9c652c","Type":"ContainerDied","Data":"6e322616bfdacb601850bafbcc5efafb9f28441701cfc12a6d0330c02ba89c00"}
Nov 02 01:54:27.218945 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:27.218914    4912 generic.go:356] "Generic (PLEG): container finished" podID="a1c8dc74-fef9-454e-90c8-b844df4403a3" containerID="64d3dc62ae593491306198c99d98098b67a4e5b46e3b7534670d341fc5475dbe" exitCode=0
Nov 02 01:54:27.237456 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:27.237426    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de06fbac-bbc3-4549-a7c2-78c10830b2c6" path="/var/lib/kubelet/pods/de06fbac-bbc3-4549-a7c2-78c10830b2c6/volumes"
Nov 02 01:54:27.237766 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:27.237749    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb" event={"ID":"a1c8dc74-fef9-454e-90c8-b844df4403a3","Type":"ContainerDied","Data":"64d3dc62ae593491306198c99d98098b67a4e5b46e3b7534670d341fc5475dbe"}
Nov 02 01:54:27.237806 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:27.237790    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb" event={"ID":"a1c8dc74-fef9-454e-90c8-b844df4403a3","Type":"ContainerStarted","Data":"d0ceb35fa8078d2360b75a1b891cde5fb17c6f25d1f06f50d95e566fbc87a384"}
Nov 02 01:54:28.428854 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.428825    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:28.480439 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.480371    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:28.503887 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.503860    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-openshift-ca\") pod \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") "
Nov 02 01:54:28.504021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.503899    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33581af1-9dd2-4f8a-9bfa-49537e9c652c-kube-api-access-rsld6\" (UniqueName: \"kubernetes.io/projected/33581af1-9dd2-4f8a-9bfa-49537e9c652c-kube-api-access-rsld6\") pod \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") "
Nov 02 01:54:28.504021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.503925    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-xks-ca\") pod \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\" (UID: \"33581af1-9dd2-4f8a-9bfa-49537e9c652c\") "
Nov 02 01:54:28.504302 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.504275    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-openshift-ca" pod "33581af1-9dd2-4f8a-9bfa-49537e9c652c" (UID: "33581af1-9dd2-4f8a-9bfa-49537e9c652c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:28.504478 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.504464    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.506210 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.506189    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-xks-ca" pod "33581af1-9dd2-4f8a-9bfa-49537e9c652c" (UID: "33581af1-9dd2-4f8a-9bfa-49537e9c652c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:28.506279 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.506203    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33581af1-9dd2-4f8a-9bfa-49537e9c652c-kube-api-access-rsld6" pod "33581af1-9dd2-4f8a-9bfa-49537e9c652c" (UID: "33581af1-9dd2-4f8a-9bfa-49537e9c652c"). InnerVolumeSpecName "kube-api-access-rsld6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:28.605590 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.605558    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c8dc74-fef9-454e-90c8-b844df4403a3-kube-api-access-n7k2l\" (UniqueName: \"kubernetes.io/projected/a1c8dc74-fef9-454e-90c8-b844df4403a3-kube-api-access-n7k2l\") pod \"a1c8dc74-fef9-454e-90c8-b844df4403a3\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") "
Nov 02 01:54:28.605805 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.605719    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-openshift-ca\") pod \"a1c8dc74-fef9-454e-90c8-b844df4403a3\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") "
Nov 02 01:54:28.605930 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.605806    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-xks-ca\") pod \"a1c8dc74-fef9-454e-90c8-b844df4403a3\" (UID: \"a1c8dc74-fef9-454e-90c8-b844df4403a3\") "
Nov 02 01:54:28.606194 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.606148    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-openshift-ca" pod "a1c8dc74-fef9-454e-90c8-b844df4403a3" (UID: "a1c8dc74-fef9-454e-90c8-b844df4403a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:28.606486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.606462    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsld6\" (UniqueName: \"kubernetes.io/projected/33581af1-9dd2-4f8a-9bfa-49537e9c652c-kube-api-access-rsld6\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.606486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.606486    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33581af1-9dd2-4f8a-9bfa-49537e9c652c-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.606669 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.606495    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.608532 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.608503    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c8dc74-fef9-454e-90c8-b844df4403a3-kube-api-access-n7k2l" pod "a1c8dc74-fef9-454e-90c8-b844df4403a3" (UID: "a1c8dc74-fef9-454e-90c8-b844df4403a3"). InnerVolumeSpecName "kube-api-access-n7k2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:28.609193 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.609169    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-xks-ca" pod "a1c8dc74-fef9-454e-90c8-b844df4403a3" (UID: "a1c8dc74-fef9-454e-90c8-b844df4403a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:28.707831 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.707791    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7k2l\" (UniqueName: \"kubernetes.io/projected/a1c8dc74-fef9-454e-90c8-b844df4403a3-kube-api-access-n7k2l\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.707831 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:28.707833    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c8dc74-fef9-454e-90c8-b844df4403a3-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:29.238581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:29.238382    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"
Nov 02 01:54:29.238581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:29.238431    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb" event={"ID":"a1c8dc74-fef9-454e-90c8-b844df4403a3","Type":"ContainerDied","Data":"d0ceb35fa8078d2360b75a1b891cde5fb17c6f25d1f06f50d95e566fbc87a384"}
Nov 02 01:54:29.238581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:29.238491    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ceb35fa8078d2360b75a1b891cde5fb17c6f25d1f06f50d95e566fbc87a384"
Nov 02 01:54:29.240115 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:29.240095    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"
Nov 02 01:54:29.240336 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:29.240108    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr" event={"ID":"33581af1-9dd2-4f8a-9bfa-49537e9c652c","Type":"ContainerDied","Data":"945d26ad5fc5fb276919008c87d1eb9b9882d97463d8c192c7877a38162f193a"}
Nov 02 01:54:29.240443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:29.240363    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945d26ad5fc5fb276919008c87d1eb9b9882d97463d8c192c7877a38162f193a"
Nov 02 01:54:30.065642 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.065611    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"]
Nov 02 01:54:30.066670 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.066648    4912 cpu_manager.go:414] "containerMap: removing container" podUID="a1c8dc74-fef9-454e-90c8-b844df4403a3" containerName="revoke-keys"
Nov 02 01:54:30.066752 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.066674    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c8dc74-fef9-454e-90c8-b844df4403a3" containerName="revoke-keys"
Nov 02 01:54:30.066752 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.066717    4912 cpu_manager.go:414] "containerMap: removing container" podUID="33581af1-9dd2-4f8a-9bfa-49537e9c652c" containerName="revoke-keys"
Nov 02 01:54:30.066752 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.066725    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33581af1-9dd2-4f8a-9bfa-49537e9c652c" containerName="revoke-keys"
Nov 02 01:54:30.066915 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.066877    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c8dc74-fef9-454e-90c8-b844df4403a3" containerName="revoke-keys"
Nov 02 01:54:30.066915 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.066895    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="33581af1-9dd2-4f8a-9bfa-49537e9c652c" containerName="revoke-keys"
Nov 02 01:54:30.071810 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.071790    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.124026 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.123994    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"]
Nov 02 01:54:30.172080 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.168333    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:30.172080 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.168418    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:30.227750 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.227718    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvkm\" (UniqueName: \"kubernetes.io/projected/c0d5c846-676b-4baf-94dc-eb1311c002a3-kube-api-access-rrvkm\") pod \"maas-api-revoke-sub-805bec55-jlvvf\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.227925 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.227784    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805bec55-jlvvf\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.227925 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.227828    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805bec55-jlvvf\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.329509 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.329414    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvkm\" (UniqueName: \"kubernetes.io/projected/c0d5c846-676b-4baf-94dc-eb1311c002a3-kube-api-access-rrvkm\") pod \"maas-api-revoke-sub-805bec55-jlvvf\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.329509 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.329500    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805bec55-jlvvf\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.329747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.329525    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805bec55-jlvvf\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.330195 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.330170    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805bec55-jlvvf\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.332149 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.332128    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805bec55-jlvvf\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.389770 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.389733    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvkm\" (UniqueName: \"kubernetes.io/projected/c0d5c846-676b-4baf-94dc-eb1311c002a3-kube-api-access-rrvkm\") pod \"maas-api-revoke-sub-805bec55-jlvvf\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.394740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.394714    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:30.542124 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.542089    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:30.786899 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:30.786869    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"]
Nov 02 01:54:30.787838 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:30.787807    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d5c846_676b_4baf_94dc_eb1311c002a3.slice/crio-dfa5f0414ef1bc4bf7d648bb2b04ae9320a70820cf15cbdcb6bb9c0c4cc191b2 WatchSource:0}: Error finding container dfa5f0414ef1bc4bf7d648bb2b04ae9320a70820cf15cbdcb6bb9c0c4cc191b2: Status 404 returned error can't find the container with id dfa5f0414ef1bc4bf7d648bb2b04ae9320a70820cf15cbdcb6bb9c0c4cc191b2
Nov 02 01:54:31.262662 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:31.262628    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf" event={"ID":"c0d5c846-676b-4baf-94dc-eb1311c002a3","Type":"ContainerStarted","Data":"670d6c04552d20fca8912019038e491d9226fac0a7ffdf4a7cfe3b178cb712c8"}
Nov 02 01:54:31.263339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:31.262681    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf" event={"ID":"c0d5c846-676b-4baf-94dc-eb1311c002a3","Type":"ContainerStarted","Data":"dfa5f0414ef1bc4bf7d648bb2b04ae9320a70820cf15cbdcb6bb9c0c4cc191b2"}
Nov 02 01:54:31.381513 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:31.381433    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf" podStartSLOduration=1.38141615 podStartE2EDuration="1.38141615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:31.376714048 +0000 UTC m=+1364.282179406" watchObservedRunningTime="2026-11-02 01:54:31.38141615 +0000 UTC m=+1364.286881507"
Nov 02 01:54:32.273161 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:32.273129    4912 generic.go:356] "Generic (PLEG): container finished" podID="c0d5c846-676b-4baf-94dc-eb1311c002a3" containerID="670d6c04552d20fca8912019038e491d9226fac0a7ffdf4a7cfe3b178cb712c8" exitCode=0
Nov 02 01:54:32.273597 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:32.273211    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf" event={"ID":"c0d5c846-676b-4baf-94dc-eb1311c002a3","Type":"ContainerDied","Data":"670d6c04552d20fca8912019038e491d9226fac0a7ffdf4a7cfe3b178cb712c8"}
Nov 02 01:54:33.460825 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.460797    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:33.577032 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.576948    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-xks-ca\") pod \"c0d5c846-676b-4baf-94dc-eb1311c002a3\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") "
Nov 02 01:54:33.577227 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.577046    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-openshift-ca\") pod \"c0d5c846-676b-4baf-94dc-eb1311c002a3\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") "
Nov 02 01:54:33.577285 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.577228    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d5c846-676b-4baf-94dc-eb1311c002a3-kube-api-access-rrvkm\" (UniqueName: \"kubernetes.io/projected/c0d5c846-676b-4baf-94dc-eb1311c002a3-kube-api-access-rrvkm\") pod \"c0d5c846-676b-4baf-94dc-eb1311c002a3\" (UID: \"c0d5c846-676b-4baf-94dc-eb1311c002a3\") "
Nov 02 01:54:33.577546 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.577500    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-openshift-ca" pod "c0d5c846-676b-4baf-94dc-eb1311c002a3" (UID: "c0d5c846-676b-4baf-94dc-eb1311c002a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:33.578108 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.578085    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.580413 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.580347    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-xks-ca" pod "c0d5c846-676b-4baf-94dc-eb1311c002a3" (UID: "c0d5c846-676b-4baf-94dc-eb1311c002a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:33.583678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.583649    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d5c846-676b-4baf-94dc-eb1311c002a3-kube-api-access-rrvkm" pod "c0d5c846-676b-4baf-94dc-eb1311c002a3" (UID: "c0d5c846-676b-4baf-94dc-eb1311c002a3"). InnerVolumeSpecName "kube-api-access-rrvkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:33.679491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.679455    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrvkm\" (UniqueName: \"kubernetes.io/projected/c0d5c846-676b-4baf-94dc-eb1311c002a3-kube-api-access-rrvkm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.679491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:33.679486    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d5c846-676b-4baf-94dc-eb1311c002a3-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.049650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:34.049613    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"]
Nov 02 01:54:34.054232 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:34.054209    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc4c43c-7nv82"]
Nov 02 01:54:34.291356 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:34.291330    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"
Nov 02 01:54:34.291526 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:34.291369    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf" event={"ID":"c0d5c846-676b-4baf-94dc-eb1311c002a3","Type":"ContainerDied","Data":"dfa5f0414ef1bc4bf7d648bb2b04ae9320a70820cf15cbdcb6bb9c0c4cc191b2"}
Nov 02 01:54:34.291526 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:34.291416    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa5f0414ef1bc4bf7d648bb2b04ae9320a70820cf15cbdcb6bb9c0c4cc191b2"
Nov 02 01:54:35.230383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:35.230352    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06eee3ae-1121-44f4-af47-4b47a2dd0ab3" path="/var/lib/kubelet/pods/06eee3ae-1121-44f4-af47-4b47a2dd0ab3/volumes"
Nov 02 01:54:35.258310 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:35.258278    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:40.109995 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.109961    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"]
Nov 02 01:54:40.110850 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.110830    4912 cpu_manager.go:414] "containerMap: removing container" podUID="c0d5c846-676b-4baf-94dc-eb1311c002a3" containerName="revoke-keys"
Nov 02 01:54:40.110955 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.110854    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d5c846-676b-4baf-94dc-eb1311c002a3" containerName="revoke-keys"
Nov 02 01:54:40.111121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.111106    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d5c846-676b-4baf-94dc-eb1311c002a3" containerName="revoke-keys"
Nov 02 01:54:40.115120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.115099    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.168415 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.168374    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549dde4d-6wtvq\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.168713 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.168686    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czd96\" (UniqueName: \"kubernetes.io/projected/7ab59dd4-09e3-4be0-8ea3-843f59f97244-kube-api-access-czd96\") pod \"maas-api-revoke-sub-549dde4d-6wtvq\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.169163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.168866    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549dde4d-6wtvq\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.169707 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.169687    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:40.174031 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.174010    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"]
Nov 02 01:54:40.195574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.195540    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:40.225356 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.225265    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:40.270198 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.270161    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549dde4d-6wtvq\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.270408 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.270224    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czd96\" (UniqueName: \"kubernetes.io/projected/7ab59dd4-09e3-4be0-8ea3-843f59f97244-kube-api-access-czd96\") pod \"maas-api-revoke-sub-549dde4d-6wtvq\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.270535 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.270515    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549dde4d-6wtvq\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.271167 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.271144    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549dde4d-6wtvq\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.272743 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.272715    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549dde4d-6wtvq\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.331704 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.331679    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czd96\" (UniqueName: \"kubernetes.io/projected/7ab59dd4-09e3-4be0-8ea3-843f59f97244-kube-api-access-czd96\") pod \"maas-api-revoke-sub-549dde4d-6wtvq\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.429812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.429772    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:40.773082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.772986    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7025-387217"]
Nov 02 01:54:40.778584 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.778560    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7025-387217"
Nov 02 01:54:40.831044 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.831015    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v4n9t\""
Nov 02 01:54:40.831890 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.831721    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"]
Nov 02 01:54:40.832166 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:40.832146    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab59dd4_09e3_4be0_8ea3_843f59f97244.slice/crio-317c273971846d68cde6a00d669aae5aa2bd14b3d43d1ee40986254082e1263f WatchSource:0}: Error finding container 317c273971846d68cde6a00d669aae5aa2bd14b3d43d1ee40986254082e1263f: Status 404 returned error can't find the container with id 317c273971846d68cde6a00d669aae5aa2bd14b3d43d1ee40986254082e1263f
Nov 02 01:54:40.838282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.838260    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7025-387217"]
Nov 02 01:54:40.878218 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.878184    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7w5d\" (UniqueName: \"kubernetes.io/projected/090c67cd-e3e2-4a75-8eb4-8a7d5056ea69-kube-api-access-m7w5d\") pod \"test-curl-7025-387217\" (UID: \"090c67cd-e3e2-4a75-8eb4-8a7d5056ea69\") " pod="opendatahub/test-curl-7025-387217"
Nov 02 01:54:40.979665 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:40.979624    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7w5d\" (UniqueName: \"kubernetes.io/projected/090c67cd-e3e2-4a75-8eb4-8a7d5056ea69-kube-api-access-m7w5d\") pod \"test-curl-7025-387217\" (UID: \"090c67cd-e3e2-4a75-8eb4-8a7d5056ea69\") " pod="opendatahub/test-curl-7025-387217"
Nov 02 01:54:41.040633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.040564    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7w5d\" (UniqueName: \"kubernetes.io/projected/090c67cd-e3e2-4a75-8eb4-8a7d5056ea69-kube-api-access-m7w5d\") pod \"test-curl-7025-387217\" (UID: \"090c67cd-e3e2-4a75-8eb4-8a7d5056ea69\") " pod="opendatahub/test-curl-7025-387217"
Nov 02 01:54:41.092704 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.092669    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7025-387217"
Nov 02 01:54:41.384472 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.384435    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq" event={"ID":"7ab59dd4-09e3-4be0-8ea3-843f59f97244","Type":"ContainerStarted","Data":"1456a90f956c03bcb8bafcb66784ba4f37978f93802b1da3903325392f60fa9b"}
Nov 02 01:54:41.384975 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.384495    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq" event={"ID":"7ab59dd4-09e3-4be0-8ea3-843f59f97244","Type":"ContainerStarted","Data":"317c273971846d68cde6a00d669aae5aa2bd14b3d43d1ee40986254082e1263f"}
Nov 02 01:54:41.480600 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:41.480557    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod090c67cd_e3e2_4a75_8eb4_8a7d5056ea69.slice/crio-700e11cbb2dc8f6fad1e3707831dc3d26d61388857a16be65284a9c3dfb003f0 WatchSource:0}: Error finding container 700e11cbb2dc8f6fad1e3707831dc3d26d61388857a16be65284a9c3dfb003f0: Status 404 returned error can't find the container with id 700e11cbb2dc8f6fad1e3707831dc3d26d61388857a16be65284a9c3dfb003f0
Nov 02 01:54:41.481042 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.481020    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7025-387217"]
Nov 02 01:54:41.482528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.482510    4912 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:54:41.500900 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.500843    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq" podStartSLOduration=1.500829155 podStartE2EDuration="1.500829155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:41.498227944 +0000 UTC m=+1374.403693303" watchObservedRunningTime="2026-11-02 01:54:41.500829155 +0000 UTC m=+1374.406294529"
Nov 02 01:54:41.914617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.914580    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"]
Nov 02 01:54:41.919860 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.919830    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:41.972959 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.972928    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"]
Nov 02 01:54:41.996737 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.996705    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0483919d-rg79k\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:41.997028 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.997003    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0483919d-rg79k\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:41.997143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:41.997047    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rfn\" (UniqueName: \"kubernetes.io/projected/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-kube-api-access-n5rfn\") pod \"maas-api-revoke-sub-0483919d-rg79k\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:42.052248 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.052211    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"]
Nov 02 01:54:42.087791 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.087759    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec82e0e-g8nvr"]
Nov 02 01:54:42.097949 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.097920    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0483919d-rg79k\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:42.098139 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.097954    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rfn\" (UniqueName: \"kubernetes.io/projected/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-kube-api-access-n5rfn\") pod \"maas-api-revoke-sub-0483919d-rg79k\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:42.098139 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.097999    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0483919d-rg79k\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:42.098775 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.098709    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0483919d-rg79k\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:42.100569 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.100547    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0483919d-rg79k\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:42.142572 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:54:42.142474    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:54:42.158954 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.158927    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rfn\" (UniqueName: \"kubernetes.io/projected/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-kube-api-access-n5rfn\") pod \"maas-api-revoke-sub-0483919d-rg79k\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:42.235090 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.235001    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:42.396893 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.396860    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7025-387217" event={"ID":"090c67cd-e3e2-4a75-8eb4-8a7d5056ea69","Type":"ContainerStarted","Data":"700e11cbb2dc8f6fad1e3707831dc3d26d61388857a16be65284a9c3dfb003f0"}
Nov 02 01:54:42.398837 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.398811    4912 generic.go:356] "Generic (PLEG): container finished" podID="7ab59dd4-09e3-4be0-8ea3-843f59f97244" containerID="1456a90f956c03bcb8bafcb66784ba4f37978f93802b1da3903325392f60fa9b" exitCode=0
Nov 02 01:54:42.398972 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.398878    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq" event={"ID":"7ab59dd4-09e3-4be0-8ea3-843f59f97244","Type":"ContainerDied","Data":"1456a90f956c03bcb8bafcb66784ba4f37978f93802b1da3903325392f60fa9b"}
Nov 02 01:54:42.621975 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:42.621942    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"]
Nov 02 01:54:42.622564 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:42.622536    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8680e8_d498_4df1_89d2_ab7d69fca0f4.slice/crio-eb727fc55964b337043612f259cb161e457d9b2f4073e56cc7fa3f03aeff550d WatchSource:0}: Error finding container eb727fc55964b337043612f259cb161e457d9b2f4073e56cc7fa3f03aeff550d: Status 404 returned error can't find the container with id eb727fc55964b337043612f259cb161e457d9b2f4073e56cc7fa3f03aeff550d
Nov 02 01:54:43.232926 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.232882    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027f1ebf-b2bc-4f43-a9ce-de3cce41157d" path="/var/lib/kubelet/pods/027f1ebf-b2bc-4f43-a9ce-de3cce41157d/volumes"
Nov 02 01:54:43.412273 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.412238    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k" event={"ID":"5f8680e8-d498-4df1-89d2-ab7d69fca0f4","Type":"ContainerStarted","Data":"fc65840f91daa5642e6b2b1240f2d56c50d5d19675e0a052123082d8acf8bc52"}
Nov 02 01:54:43.412715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.412298    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k" event={"ID":"5f8680e8-d498-4df1-89d2-ab7d69fca0f4","Type":"ContainerStarted","Data":"eb727fc55964b337043612f259cb161e457d9b2f4073e56cc7fa3f03aeff550d"}
Nov 02 01:54:43.535922 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.535345    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k" podStartSLOduration=2.535325677 podStartE2EDuration="2.535325677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:43.530441594 +0000 UTC m=+1376.435906952" watchObservedRunningTime="2026-11-02 01:54:43.535325677 +0000 UTC m=+1376.440791034"
Nov 02 01:54:43.654375 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.654347    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:43.724670 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.724641    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab59dd4-09e3-4be0-8ea3-843f59f97244-kube-api-access-czd96\" (UniqueName: \"kubernetes.io/projected/7ab59dd4-09e3-4be0-8ea3-843f59f97244-kube-api-access-czd96\") pod \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") "
Nov 02 01:54:43.724841 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.724726    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-xks-ca\") pod \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") "
Nov 02 01:54:43.724841 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.724774    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-openshift-ca\") pod \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\" (UID: \"7ab59dd4-09e3-4be0-8ea3-843f59f97244\") "
Nov 02 01:54:43.725392 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.725362    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-openshift-ca" pod "7ab59dd4-09e3-4be0-8ea3-843f59f97244" (UID: "7ab59dd4-09e3-4be0-8ea3-843f59f97244"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:43.727964 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.727933    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-xks-ca" pod "7ab59dd4-09e3-4be0-8ea3-843f59f97244" (UID: "7ab59dd4-09e3-4be0-8ea3-843f59f97244"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:43.728210 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.728189    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab59dd4-09e3-4be0-8ea3-843f59f97244-kube-api-access-czd96" pod "7ab59dd4-09e3-4be0-8ea3-843f59f97244" (UID: "7ab59dd4-09e3-4be0-8ea3-843f59f97244"). InnerVolumeSpecName "kube-api-access-czd96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:43.827673 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.827634    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czd96\" (UniqueName: \"kubernetes.io/projected/7ab59dd4-09e3-4be0-8ea3-843f59f97244-kube-api-access-czd96\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:43.827673 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.827675    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:43.827930 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:43.827691    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab59dd4-09e3-4be0-8ea3-843f59f97244-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:44.425974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:44.425937    4912 generic.go:356] "Generic (PLEG): container finished" podID="5f8680e8-d498-4df1-89d2-ab7d69fca0f4" containerID="fc65840f91daa5642e6b2b1240f2d56c50d5d19675e0a052123082d8acf8bc52" exitCode=0
Nov 02 01:54:44.426447 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:44.426026    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k" event={"ID":"5f8680e8-d498-4df1-89d2-ab7d69fca0f4","Type":"ContainerDied","Data":"fc65840f91daa5642e6b2b1240f2d56c50d5d19675e0a052123082d8acf8bc52"}
Nov 02 01:54:44.428222 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:44.428180    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"
Nov 02 01:54:44.428349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:44.428217    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq" event={"ID":"7ab59dd4-09e3-4be0-8ea3-843f59f97244","Type":"ContainerDied","Data":"317c273971846d68cde6a00d669aae5aa2bd14b3d43d1ee40986254082e1263f"}
Nov 02 01:54:44.428349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:44.428264    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317c273971846d68cde6a00d669aae5aa2bd14b3d43d1ee40986254082e1263f"
Nov 02 01:54:45.187134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.187100    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:45.415571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.415535    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:45.438695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.438603    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7025-387217" event={"ID":"090c67cd-e3e2-4a75-8eb4-8a7d5056ea69","Type":"ContainerStarted","Data":"b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12"}
Nov 02 01:54:45.560772 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.558311    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7025-387217" podStartSLOduration=2.392646935 podStartE2EDuration="5.558289234s" totalImagesPullingTime="3.165642299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:45.552196567 +0000 UTC m=+1378.457661932" watchObservedRunningTime="2026-11-02 01:54:45.558289234 +0000 UTC m=+1378.463754598"
Nov 02 01:54:45.630750 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.630725    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:45.753362 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.753272    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-openshift-ca\") pod \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") "
Nov 02 01:54:45.753362 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.753315    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-kube-api-access-n5rfn\" (UniqueName: \"kubernetes.io/projected/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-kube-api-access-n5rfn\") pod \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") "
Nov 02 01:54:45.753674 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.753649    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-xks-ca\") pod \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\" (UID: \"5f8680e8-d498-4df1-89d2-ab7d69fca0f4\") "
Nov 02 01:54:45.754038 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.753651    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-openshift-ca" pod "5f8680e8-d498-4df1-89d2-ab7d69fca0f4" (UID: "5f8680e8-d498-4df1-89d2-ab7d69fca0f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:45.754856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.754831    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:45.756298 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.756272    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-xks-ca" pod "5f8680e8-d498-4df1-89d2-ab7d69fca0f4" (UID: "5f8680e8-d498-4df1-89d2-ab7d69fca0f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:45.756528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.756499    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-kube-api-access-n5rfn" pod "5f8680e8-d498-4df1-89d2-ab7d69fca0f4" (UID: "5f8680e8-d498-4df1-89d2-ab7d69fca0f4"). InnerVolumeSpecName "kube-api-access-n5rfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:45.856228 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.856200    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:45.856228 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.856226    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rfn\" (UniqueName: \"kubernetes.io/projected/5f8680e8-d498-4df1-89d2-ab7d69fca0f4-kube-api-access-n5rfn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:45.952129 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.952095    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"]
Nov 02 01:54:45.952917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.952895    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7ab59dd4-09e3-4be0-8ea3-843f59f97244" containerName="revoke-keys"
Nov 02 01:54:45.952917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.952916    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab59dd4-09e3-4be0-8ea3-843f59f97244" containerName="revoke-keys"
Nov 02 01:54:45.953136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.952959    4912 cpu_manager.go:414] "containerMap: removing container" podUID="5f8680e8-d498-4df1-89d2-ab7d69fca0f4" containerName="revoke-keys"
Nov 02 01:54:45.953136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.952968    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f8680e8-d498-4df1-89d2-ab7d69fca0f4" containerName="revoke-keys"
Nov 02 01:54:45.953243 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.953156    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab59dd4-09e3-4be0-8ea3-843f59f97244" containerName="revoke-keys"
Nov 02 01:54:45.953243 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.953178    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f8680e8-d498-4df1-89d2-ab7d69fca0f4" containerName="revoke-keys"
Nov 02 01:54:45.957140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:45.957117    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.012597 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.012518    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"]
Nov 02 01:54:46.058617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.058586    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg59x\" (UniqueName: \"kubernetes.io/projected/4f02df9b-e86f-4bbc-b93d-624dde3296e0-kube-api-access-pg59x\") pod \"maas-api-revoke-sub-a237b1fa-5dkbw\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.058820 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.058655    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a237b1fa-5dkbw\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.058868 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.058810    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a237b1fa-5dkbw\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.160603 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.160564    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a237b1fa-5dkbw\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.160799 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.160780    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg59x\" (UniqueName: \"kubernetes.io/projected/4f02df9b-e86f-4bbc-b93d-624dde3296e0-kube-api-access-pg59x\") pod \"maas-api-revoke-sub-a237b1fa-5dkbw\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.160864 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.160845    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a237b1fa-5dkbw\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.161729 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.161675    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a237b1fa-5dkbw\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.163828 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.163801    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a237b1fa-5dkbw\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.221978 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.221949    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg59x\" (UniqueName: \"kubernetes.io/projected/4f02df9b-e86f-4bbc-b93d-624dde3296e0-kube-api-access-pg59x\") pod \"maas-api-revoke-sub-a237b1fa-5dkbw\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.274865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.274773    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:46.449763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.449734    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"
Nov 02 01:54:46.450206 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.449756    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k" event={"ID":"5f8680e8-d498-4df1-89d2-ab7d69fca0f4","Type":"ContainerDied","Data":"eb727fc55964b337043612f259cb161e457d9b2f4073e56cc7fa3f03aeff550d"}
Nov 02 01:54:46.450206 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.449815    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb727fc55964b337043612f259cb161e457d9b2f4073e56cc7fa3f03aeff550d"
Nov 02 01:54:46.686949 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:46.686918    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f02df9b_e86f_4bbc_b93d_624dde3296e0.slice/crio-fe534e13c122a789a2e8ad72dfde756359e678bfc54fb7d01a3b373de017683d WatchSource:0}: Error finding container fe534e13c122a789a2e8ad72dfde756359e678bfc54fb7d01a3b373de017683d: Status 404 returned error can't find the container with id fe534e13c122a789a2e8ad72dfde756359e678bfc54fb7d01a3b373de017683d
Nov 02 01:54:46.688297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:46.688273    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"]
Nov 02 01:54:47.066974 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.066944    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7025-387217"]
Nov 02 01:54:47.082142 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.074366    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7025-387217"]
Nov 02 01:54:47.215357 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.215324    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7025-dc544f"]
Nov 02 01:54:47.216101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.216082    4912 cpu_manager.go:414] "containerMap: removing container" podUID="090c67cd-e3e2-4a75-8eb4-8a7d5056ea69" containerName="test-curl-7025-387217"
Nov 02 01:54:47.216101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.216101    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090c67cd-e3e2-4a75-8eb4-8a7d5056ea69" containerName="test-curl-7025-387217"
Nov 02 01:54:47.216288 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.216273    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="090c67cd-e3e2-4a75-8eb4-8a7d5056ea69" containerName="test-curl-7025-387217"
Nov 02 01:54:47.223048 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.223022    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7025-dc544f"
Nov 02 01:54:47.273796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.273748    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7025-dc544f"]
Nov 02 01:54:47.280692 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.280291    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdtl\" (UniqueName: \"kubernetes.io/projected/f2162a8e-aa87-48f4-bb13-1672ba05773e-kube-api-access-hjdtl\") pod \"test-curl-7025-dc544f\" (UID: \"f2162a8e-aa87-48f4-bb13-1672ba05773e\") " pod="opendatahub/test-curl-7025-dc544f"
Nov 02 01:54:47.382472 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.382441    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdtl\" (UniqueName: \"kubernetes.io/projected/f2162a8e-aa87-48f4-bb13-1672ba05773e-kube-api-access-hjdtl\") pod \"test-curl-7025-dc544f\" (UID: \"f2162a8e-aa87-48f4-bb13-1672ba05773e\") " pod="opendatahub/test-curl-7025-dc544f"
Nov 02 01:54:47.443376 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.443349    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdtl\" (UniqueName: \"kubernetes.io/projected/f2162a8e-aa87-48f4-bb13-1672ba05773e-kube-api-access-hjdtl\") pod \"test-curl-7025-dc544f\" (UID: \"f2162a8e-aa87-48f4-bb13-1672ba05773e\") " pod="opendatahub/test-curl-7025-dc544f"
Nov 02 01:54:47.465108 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.465035    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw" event={"ID":"4f02df9b-e86f-4bbc-b93d-624dde3296e0","Type":"ContainerStarted","Data":"8fd744c58ead86ff01f9e4649f8e99e6dbe256bfe597bdd5f0f250221078c332"}
Nov 02 01:54:47.465547 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.465144    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw" event={"ID":"4f02df9b-e86f-4bbc-b93d-624dde3296e0","Type":"ContainerStarted","Data":"fe534e13c122a789a2e8ad72dfde756359e678bfc54fb7d01a3b373de017683d"}
Nov 02 01:54:47.465547 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.465387    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7025-387217" podUID="090c67cd-e3e2-4a75-8eb4-8a7d5056ea69" containerName="test-curl-7025-387217" containerID="cri-o://b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12" gracePeriod=2
Nov 02 01:54:47.475277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.475248    4912 scope.go:122] "RemoveContainer" containerID="d7d3438b9f35ad6d7a124bad48bc2395cbae0013e70dcece55f6bf8789de2e54"
Nov 02 01:54:47.486486 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.486460    4912 scope.go:122] "RemoveContainer" containerID="8346520c01dac7bb4fc924334515ff2e63dde8c0ab5f513deac8cca20a5f96d5"
Nov 02 01:54:47.498906 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.498878    4912 scope.go:122] "RemoveContainer" containerID="4b80cc2e20415e90ab7568a8d05c0b10641ff11ffb580dbbfc2cbf54ae130e5c"
Nov 02 01:54:47.538654 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.538634    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7025-dc544f"
Nov 02 01:54:47.583886 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.583824    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw" podStartSLOduration=2.583802571 podStartE2EDuration="2.583802571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:47.581879569 +0000 UTC m=+1380.487344922" watchObservedRunningTime="2026-11-02 01:54:47.583802571 +0000 UTC m=+1380.489267938"
Nov 02 01:54:47.629934 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.629764    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:47.857272 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.857175    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:47.939051 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:54:47.939022    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2162a8e_aa87_48f4_bb13_1672ba05773e.slice/crio-6ed77ba2d7c5b99f7af0c15c268f560128f1911caa886861e62b04c1d7f63627 WatchSource:0}: Error finding container 6ed77ba2d7c5b99f7af0c15c268f560128f1911caa886861e62b04c1d7f63627: Status 404 returned error can't find the container with id 6ed77ba2d7c5b99f7af0c15c268f560128f1911caa886861e62b04c1d7f63627
Nov 02 01:54:47.939762 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:47.939659    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7025-dc544f"]
Nov 02 01:54:48.439257 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:48.439229    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:48.476313 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:48.476276    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7025-dc544f" event={"ID":"f2162a8e-aa87-48f4-bb13-1672ba05773e","Type":"ContainerStarted","Data":"8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4"}
Nov 02 01:54:48.476805 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:48.476326    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7025-dc544f" event={"ID":"f2162a8e-aa87-48f4-bb13-1672ba05773e","Type":"ContainerStarted","Data":"6ed77ba2d7c5b99f7af0c15c268f560128f1911caa886861e62b04c1d7f63627"}
Nov 02 01:54:48.478161 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:48.478134    4912 generic.go:356] "Generic (PLEG): container finished" podID="4f02df9b-e86f-4bbc-b93d-624dde3296e0" containerID="8fd744c58ead86ff01f9e4649f8e99e6dbe256bfe597bdd5f0f250221078c332" exitCode=0
Nov 02 01:54:48.478283 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:48.478233    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw" event={"ID":"4f02df9b-e86f-4bbc-b93d-624dde3296e0","Type":"ContainerDied","Data":"8fd744c58ead86ff01f9e4649f8e99e6dbe256bfe597bdd5f0f250221078c332"}
Nov 02 01:54:48.593185 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:48.593132    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7025-dc544f" podStartSLOduration=1.5931171549999998 podStartE2EDuration="1.593117155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:48.58899689 +0000 UTC m=+1381.494462251" watchObservedRunningTime="2026-11-02 01:54:48.593117155 +0000 UTC m=+1381.498582512"
Nov 02 01:54:49.701842 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.699661    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:49.789183 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.789153    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7025-387217"
Nov 02 01:54:49.820877 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.820844    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-xks-ca\") pod \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") "
Nov 02 01:54:49.821050 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.821009    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-openshift-ca\") pod \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") "
Nov 02 01:54:49.821195 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.821177    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f02df9b-e86f-4bbc-b93d-624dde3296e0-kube-api-access-pg59x\" (UniqueName: \"kubernetes.io/projected/4f02df9b-e86f-4bbc-b93d-624dde3296e0-kube-api-access-pg59x\") pod \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\" (UID: \"4f02df9b-e86f-4bbc-b93d-624dde3296e0\") "
Nov 02 01:54:49.821435 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.821410    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-openshift-ca" pod "4f02df9b-e86f-4bbc-b93d-624dde3296e0" (UID: "4f02df9b-e86f-4bbc-b93d-624dde3296e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:49.823423 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.823395    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:49.823871 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.823839    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f02df9b-e86f-4bbc-b93d-624dde3296e0-kube-api-access-pg59x" pod "4f02df9b-e86f-4bbc-b93d-624dde3296e0" (UID: "4f02df9b-e86f-4bbc-b93d-624dde3296e0"). InnerVolumeSpecName "kube-api-access-pg59x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:49.823945 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.823923    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-xks-ca" pod "4f02df9b-e86f-4bbc-b93d-624dde3296e0" (UID: "4f02df9b-e86f-4bbc-b93d-624dde3296e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:49.924617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.924581    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090c67cd-e3e2-4a75-8eb4-8a7d5056ea69-kube-api-access-m7w5d\" (UniqueName: \"kubernetes.io/projected/090c67cd-e3e2-4a75-8eb4-8a7d5056ea69-kube-api-access-m7w5d\") pod \"090c67cd-e3e2-4a75-8eb4-8a7d5056ea69\" (UID: \"090c67cd-e3e2-4a75-8eb4-8a7d5056ea69\") "
Nov 02 01:54:49.927098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.925454    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg59x\" (UniqueName: \"kubernetes.io/projected/4f02df9b-e86f-4bbc-b93d-624dde3296e0-kube-api-access-pg59x\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:49.927098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.925499    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f02df9b-e86f-4bbc-b93d-624dde3296e0-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:49.928468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.928440    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090c67cd-e3e2-4a75-8eb4-8a7d5056ea69-kube-api-access-m7w5d" pod "090c67cd-e3e2-4a75-8eb4-8a7d5056ea69" (UID: "090c67cd-e3e2-4a75-8eb4-8a7d5056ea69"). InnerVolumeSpecName "kube-api-access-m7w5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:49.955559 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:49.955514    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:50.027624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.027515    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7w5d\" (UniqueName: \"kubernetes.io/projected/090c67cd-e3e2-4a75-8eb4-8a7d5056ea69-kube-api-access-m7w5d\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.052642 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.052600    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"]
Nov 02 01:54:50.060744 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.060703    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfda3fd-zmmn2"]
Nov 02 01:54:50.105141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.105107    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7025-dc544f"]
Nov 02 01:54:50.113040 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.112986    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7025-dc544f"]
Nov 02 01:54:50.504913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.504877    4912 generic.go:356] "Generic (PLEG): container finished" podID="090c67cd-e3e2-4a75-8eb4-8a7d5056ea69" containerID="b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12" exitCode=137
Nov 02 01:54:50.505132 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.504917    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7025-387217"
Nov 02 01:54:50.505263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.505245    4912 scope.go:122] "RemoveContainer" containerID="b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12"
Nov 02 01:54:50.507477 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.507446    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7025-dc544f" podUID="f2162a8e-aa87-48f4-bb13-1672ba05773e" containerName="test-curl-7025-dc544f" containerID="cri-o://8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4" gracePeriod=2
Nov 02 01:54:50.507871 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.507851    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"
Nov 02 01:54:50.508725 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.508697    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw" event={"ID":"4f02df9b-e86f-4bbc-b93d-624dde3296e0","Type":"ContainerDied","Data":"fe534e13c122a789a2e8ad72dfde756359e678bfc54fb7d01a3b373de017683d"}
Nov 02 01:54:50.508817 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.508771    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe534e13c122a789a2e8ad72dfde756359e678bfc54fb7d01a3b373de017683d"
Nov 02 01:54:50.520198 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.520174    4912 scope.go:122] "RemoveContainer" containerID="b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12"
Nov 02 01:54:50.520573 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:54:50.520550    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12\": container with ID starting with b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12 not found: ID does not exist" containerID="b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12"
Nov 02 01:54:50.520655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.520582    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12"} err="failed to get container status \"b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12\": rpc error: code = NotFound desc = could not find container \"b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12\": container with ID starting with b9a1781f8112ba8de03539be23b2312a46395394a399922500bec07e146b9a12 not found: ID does not exist"
Nov 02 01:54:50.560793 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:54:50.560734    4912 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7025-dc544f\" is forbidden: User \"system:node:ip-10-0-135-33.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-33.us-west-1.compute.internal' and this object" podUID="f2162a8e-aa87-48f4-bb13-1672ba05773e" pod="opendatahub/test-curl-7025-dc544f"
Nov 02 01:54:50.611692 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:54:50.611651    4912 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7025-387217\" is forbidden: User \"system:node:ip-10-0-135-33.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-33.us-west-1.compute.internal' and this object" podUID="090c67cd-e3e2-4a75-8eb4-8a7d5056ea69" pod="opendatahub/test-curl-7025-387217"
Nov 02 01:54:50.662466 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:54:50.662408    4912 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7025-dc544f\" is forbidden: User \"system:node:ip-10-0-135-33.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-33.us-west-1.compute.internal' and this object" podUID="f2162a8e-aa87-48f4-bb13-1672ba05773e" pod="opendatahub/test-curl-7025-dc544f"
Nov 02 01:54:50.929255 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:50.929212    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:51.227710 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:51.227633    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090c67cd-e3e2-4a75-8eb4-8a7d5056ea69" path="/var/lib/kubelet/pods/090c67cd-e3e2-4a75-8eb4-8a7d5056ea69/volumes"
Nov 02 01:54:51.228237 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:51.228219    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="988cd1dc-c4db-47fd-934d-d19147ecd644" path="/var/lib/kubelet/pods/988cd1dc-c4db-47fd-934d-d19147ecd644/volumes"
Nov 02 01:54:51.658449 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:51.652318    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:51.915643 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:51.915545    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:52.792655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:52.792626    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7025-dc544f"
Nov 02 01:54:52.881603 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:52.881569    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2162a8e-aa87-48f4-bb13-1672ba05773e-kube-api-access-hjdtl\" (UniqueName: \"kubernetes.io/projected/f2162a8e-aa87-48f4-bb13-1672ba05773e-kube-api-access-hjdtl\") pod \"f2162a8e-aa87-48f4-bb13-1672ba05773e\" (UID: \"f2162a8e-aa87-48f4-bb13-1672ba05773e\") "
Nov 02 01:54:52.884168 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:52.884135    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2162a8e-aa87-48f4-bb13-1672ba05773e-kube-api-access-hjdtl" pod "f2162a8e-aa87-48f4-bb13-1672ba05773e" (UID: "f2162a8e-aa87-48f4-bb13-1672ba05773e"). InnerVolumeSpecName "kube-api-access-hjdtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:52.983944 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:52.983907    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjdtl\" (UniqueName: \"kubernetes.io/projected/f2162a8e-aa87-48f4-bb13-1672ba05773e-kube-api-access-hjdtl\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:53.227895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:53.227860    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2162a8e-aa87-48f4-bb13-1672ba05773e" path="/var/lib/kubelet/pods/f2162a8e-aa87-48f4-bb13-1672ba05773e/volumes"
Nov 02 01:54:53.535488 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:53.535385    4912 generic.go:356] "Generic (PLEG): container finished" podID="f2162a8e-aa87-48f4-bb13-1672ba05773e" containerID="8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4" exitCode=137
Nov 02 01:54:53.535488 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:53.535441    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7025-dc544f"
Nov 02 01:54:53.535488 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:53.535472    4912 scope.go:122] "RemoveContainer" containerID="8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4"
Nov 02 01:54:53.546531 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:53.546505    4912 scope.go:122] "RemoveContainer" containerID="8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4"
Nov 02 01:54:53.546851 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:54:53.546833    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4\": container with ID starting with 8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4 not found: ID does not exist" containerID="8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4"
Nov 02 01:54:53.546917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:53.546861    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4"} err="failed to get container status \"8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4\": rpc error: code = NotFound desc = could not find container \"8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4\": container with ID starting with 8440acb7d4aebe558b3cfcd6e40c15d5c05e02f27945754daa2eb4fbf701f2c4 not found: ID does not exist"
Nov 02 01:54:54.344886 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:54.343689    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:54.573148 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:54.558957    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:54.755331 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:54.755283    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:55.081139 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:55.080830    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:55.299751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:55.299701    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:55.525194 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:55.525144    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:55.728081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:55.726941    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:55.946132 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:55.938113    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:56.804081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:56.802427    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:56.990082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:56.989847    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:54:57.147094 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.146820    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"]
Nov 02 01:54:57.152084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.147858    4912 cpu_manager.go:414] "containerMap: removing container" podUID="f2162a8e-aa87-48f4-bb13-1672ba05773e" containerName="test-curl-7025-dc544f"
Nov 02 01:54:57.152084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.147887    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2162a8e-aa87-48f4-bb13-1672ba05773e" containerName="test-curl-7025-dc544f"
Nov 02 01:54:57.152084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.147904    4912 cpu_manager.go:414] "containerMap: removing container" podUID="4f02df9b-e86f-4bbc-b93d-624dde3296e0" containerName="revoke-keys"
Nov 02 01:54:57.152084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.147913    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f02df9b-e86f-4bbc-b93d-624dde3296e0" containerName="revoke-keys"
Nov 02 01:54:57.152084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.148131    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f02df9b-e86f-4bbc-b93d-624dde3296e0" containerName="revoke-keys"
Nov 02 01:54:57.152084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.148168    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2162a8e-aa87-48f4-bb13-1672ba05773e" containerName="test-curl-7025-dc544f"
Nov 02 01:54:57.177611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.170243    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:57.498143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.308338    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2clt\" (UniqueName: \"kubernetes.io/projected/b44978b4-76f6-44c2-b83d-ea8c91a209a3-kube-api-access-s2clt\") pod \"maas-api-revoke-sub-4a3713e6-jf7jr\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:57.498143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.308423    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a3713e6-jf7jr\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:57.498143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.308481    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:57.498143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.308579    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a3713e6-jf7jr\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:57.498143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.361380    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 02 01:54:57.498143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.362500    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:57.498143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.456857    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2clt\" (UniqueName: \"kubernetes.io/projected/b44978b4-76f6-44c2-b83d-ea8c91a209a3-kube-api-access-s2clt\") pod \"maas-api-revoke-sub-4a3713e6-jf7jr\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:57.498143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.456929    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a3713e6-jf7jr\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:57.503623 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.498925    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a3713e6-jf7jr\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:57.538349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.538164    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a3713e6-jf7jr\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:57.594628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.309574    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:57.623016 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.622228    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:57.814371 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.814234    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:57.872588 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.872548    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:57.901711 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.901490    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 01:54:57.999084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:57.989749    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"]
Nov 02 01:54:58.065145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.030736    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a3713e6-jf7jr\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:58.078082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.074863    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:58.144783 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.144738    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Nov 02 01:54:58.170330 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.153981    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2clt\" (UniqueName: \"kubernetes.io/projected/b44978b4-76f6-44c2-b83d-ea8c91a209a3-kube-api-access-s2clt\") pod \"maas-api-revoke-sub-4a3713e6-jf7jr\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:58.170330 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.154106    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:58.170330 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.161148    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:58.199815 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.199761    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:58.351282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.351178    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:58.366562 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.366518    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:58.389567 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.366798    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 02 01:54:58.391130 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.391099    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:54:58.391431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.391416    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:54:58.408021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.403503    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:54:58.678332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.678282    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:58.678914 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:58.678888    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:54:59.024474 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.024343    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 01:54:59.025584 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.025553    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 01:54:59.026163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.026126    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 01:54:59.146097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.132979    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:54:59.146529 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.146499    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:54:59.146650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.146610    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:54:59.146773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.037082    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.156101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.148221    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 01:54:59.156101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.148769    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:54:59.173384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.173341    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.178084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.174324    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:54:59.235085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.202207    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.235085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.202758    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:54:59.235085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.146167    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.235085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.204116    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:54:59.235802 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.235746    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 01:54:59.240510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.240484    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:54:59.316650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.316543    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.378085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.374332    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:54:59.378375 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.378343    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.383085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.379260    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.431087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.405624    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.431869 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.431826    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:54:59.527576 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.432645    4912 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:54:59.527879 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.527845    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.708086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.708027    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.984478 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:54:59.984391    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p" podUID="94358ff5-ab4d-4e82-880f-515d87a4a790" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:00.184362 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.056190    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 01:55:00.192541 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.056628    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.026180 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.056676    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.058085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.238186    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.058085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.266316    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pgx5n" podUID="825d5fda-b233-458a-93f7-31f99cecadb3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.058085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.271172    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.060699 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.195138    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.079113 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.315182    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.079113 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.067970    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.079113 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:55:00.514422    4912 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.197s"
Nov 02 01:55:01.079113 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.068152    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:55:01.079113 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.584650    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg" podUID="62cc99ac-a5bc-4c99-a2e6-8487bb9f994e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.316520    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-46j9s" podUID="8df16c8c-28c7-46e5-a791-0bed28ecb05a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.704362    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded"
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.704400    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.705608    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.718269    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.741396    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.759707    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.082209    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.704297    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.113083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:00.645320    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.201473 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.201427    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.768726 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.768658    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:55:01.769012 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.768995    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:55:01.828805 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.828763    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:01.837020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.836474    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:55:01.837020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.836543    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:55:01.837020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.836661    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:55:01.837020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.836703    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:55:01.837020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.836789    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:55:01.837020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.836828    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:55:01.837020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.836858    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:55:01.837020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.836886    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:55:01.837020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.836915    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:55:01.838020 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.837240    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:55:01.843792 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.843771    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"]
Nov 02 01:55:01.866159 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.866045    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:01.882780 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.882738    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"]
Nov 02 01:55:01.892581 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.892520    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"]
Nov 02 01:55:01.905867 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.905826    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:01.912664 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.912632    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"]
Nov 02 01:55:01.920708 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.920679    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-15487453\""
Nov 02 01:55:01.920897 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.920696    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-15487453-dockercfg-m2l2b\""
Nov 02 01:55:01.933704 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.933657    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cbbb0cf-795e-4489-a161-f2002e56c3f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m\" (UID: \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\") " pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:01.933908 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.933758    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e117c74a-26e9-49a4-9429-45b68fd1a634-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk\" (UID: \"e117c74a-26e9-49a4-9429-45b68fd1a634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:01.934002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.933967    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrlhp\" (UniqueName: \"kubernetes.io/projected/2cbbb0cf-795e-4489-a161-f2002e56c3f7-kube-api-access-xrlhp\") pod \"payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m\" (UID: \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\") " pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:01.934170 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:01.934143    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr62q\" (UniqueName: \"kubernetes.io/projected/e117c74a-26e9-49a4-9429-45b68fd1a634-kube-api-access-wr62q\") pod \"payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk\" (UID: \"e117c74a-26e9-49a4-9429-45b68fd1a634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:02.018151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.018045    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:02.036721 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.036668    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr62q\" (UniqueName: \"kubernetes.io/projected/e117c74a-26e9-49a4-9429-45b68fd1a634-kube-api-access-wr62q\") pod \"payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk\" (UID: \"e117c74a-26e9-49a4-9429-45b68fd1a634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:02.036936 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.036917    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cbbb0cf-795e-4489-a161-f2002e56c3f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m\" (UID: \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\") " pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:02.036998 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.036975    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e117c74a-26e9-49a4-9429-45b68fd1a634-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk\" (UID: \"e117c74a-26e9-49a4-9429-45b68fd1a634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:02.037101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.037080    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrlhp\" (UniqueName: \"kubernetes.io/projected/2cbbb0cf-795e-4489-a161-f2002e56c3f7-kube-api-access-xrlhp\") pod \"payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m\" (UID: \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\") " pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:02.037948 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.037920    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cbbb0cf-795e-4489-a161-f2002e56c3f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m\" (UID: \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\") " pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:02.038258 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.038230    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e117c74a-26e9-49a4-9429-45b68fd1a634-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk\" (UID: \"e117c74a-26e9-49a4-9429-45b68fd1a634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:02.039630 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.039605    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"]
Nov 02 01:55:02.061245 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:02.061023    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44978b4_76f6_44c2_b83d_ea8c91a209a3.slice/crio-8fb1b8c8a734c8cf20c6f12e7b57ac5ace2960492256e0576b3f17792d964aee WatchSource:0}: Error finding container 8fb1b8c8a734c8cf20c6f12e7b57ac5ace2960492256e0576b3f17792d964aee: Status 404 returned error can't find the container with id 8fb1b8c8a734c8cf20c6f12e7b57ac5ace2960492256e0576b3f17792d964aee
Nov 02 01:55:02.098964 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.098922    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr62q\" (UniqueName: \"kubernetes.io/projected/e117c74a-26e9-49a4-9429-45b68fd1a634-kube-api-access-wr62q\") pod \"payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk\" (UID: \"e117c74a-26e9-49a4-9429-45b68fd1a634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:02.099285 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.099261    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrlhp\" (UniqueName: \"kubernetes.io/projected/2cbbb0cf-795e-4489-a161-f2002e56c3f7-kube-api-access-xrlhp\") pod \"payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m\" (UID: \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\") " pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:02.219140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.219108    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:02.229919 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.229874    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:02.746614 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:02.746584    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode117c74a_26e9_49a4_9429_45b68fd1a634.slice/crio-356bb3adc8c3421e247ef7273b7acc0de720b46aa6d6f3e1a46b6eb390174437 WatchSource:0}: Error finding container 356bb3adc8c3421e247ef7273b7acc0de720b46aa6d6f3e1a46b6eb390174437: Status 404 returned error can't find the container with id 356bb3adc8c3421e247ef7273b7acc0de720b46aa6d6f3e1a46b6eb390174437
Nov 02 01:55:02.746758 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.746715    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"]
Nov 02 01:55:02.783969 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.783941    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"]
Nov 02 01:55:02.795681 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:02.795647    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbbb0cf_795e_4489_a161_f2002e56c3f7.slice/crio-c70b0ddce208404544d62b118a26066a922663d4925fa37985fa4c5d0ded18c2 WatchSource:0}: Error finding container c70b0ddce208404544d62b118a26066a922663d4925fa37985fa4c5d0ded18c2: Status 404 returned error can't find the container with id c70b0ddce208404544d62b118a26066a922663d4925fa37985fa4c5d0ded18c2
Nov 02 01:55:02.882622 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.882542    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m" event={"ID":"2cbbb0cf-795e-4489-a161-f2002e56c3f7","Type":"ContainerStarted","Data":"c70b0ddce208404544d62b118a26066a922663d4925fa37985fa4c5d0ded18c2"}
Nov 02 01:55:02.885436 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.885407    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr" event={"ID":"b44978b4-76f6-44c2-b83d-ea8c91a209a3","Type":"ContainerStarted","Data":"df8e2f95b3045aadae0301f85de201fdb7ab9464f29953e83ac68df389764fa2"}
Nov 02 01:55:02.890234 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.890189    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr" event={"ID":"b44978b4-76f6-44c2-b83d-ea8c91a209a3","Type":"ContainerStarted","Data":"8fb1b8c8a734c8cf20c6f12e7b57ac5ace2960492256e0576b3f17792d964aee"}
Nov 02 01:55:02.890478 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.890443    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:02.890637 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.890618    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk" event={"ID":"e117c74a-26e9-49a4-9429-45b68fd1a634","Type":"ContainerStarted","Data":"18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743"}
Nov 02 01:55:02.890786 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.890771    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk" event={"ID":"e117c74a-26e9-49a4-9429-45b68fd1a634","Type":"ContainerStarted","Data":"356bb3adc8c3421e247ef7273b7acc0de720b46aa6d6f3e1a46b6eb390174437"}
Nov 02 01:55:02.973773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:02.973737    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:03.233612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:03.233581    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:03.418348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:03.418243    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:03.905734 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:03.905700    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m" event={"ID":"2cbbb0cf-795e-4489-a161-f2002e56c3f7","Type":"ContainerStarted","Data":"b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c"}
Nov 02 01:55:03.905958 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:03.905804    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:03.907662 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:03.907639    4912 generic.go:356] "Generic (PLEG): container finished" podID="b44978b4-76f6-44c2-b83d-ea8c91a209a3" containerID="df8e2f95b3045aadae0301f85de201fdb7ab9464f29953e83ac68df389764fa2" exitCode=0
Nov 02 01:55:03.907936 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:03.907908    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr" event={"ID":"b44978b4-76f6-44c2-b83d-ea8c91a209a3","Type":"ContainerDied","Data":"df8e2f95b3045aadae0301f85de201fdb7ab9464f29953e83ac68df389764fa2"}
Nov 02 01:55:03.994187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:03.994130    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk" podStartSLOduration=3.994112835 podStartE2EDuration="3.994112835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:03.993330143 +0000 UTC m=+1396.898795511" watchObservedRunningTime="2026-11-02 01:55:03.994112835 +0000 UTC m=+1396.899578192"
Nov 02 01:55:03.994853 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:03.994808    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr" podStartSLOduration=6.994797203 podStartE2EDuration="6.994797203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:03.878409919 +0000 UTC m=+1396.783875279" watchObservedRunningTime="2026-11-02 01:55:03.994797203 +0000 UTC m=+1396.900262560"
Nov 02 01:55:04.226856 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:04.226751    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m" podStartSLOduration=3.226734154 podStartE2EDuration="3.226734154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:04.223866415 +0000 UTC m=+1397.129331771" watchObservedRunningTime="2026-11-02 01:55:04.226734154 +0000 UTC m=+1397.132199510"
Nov 02 01:55:04.736998 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:04.736955    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:05.112934 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.112902    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:55:05.206764 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.206726    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-openshift-ca\") pod \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") "
Nov 02 01:55:05.206969 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.206806    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44978b4-76f6-44c2-b83d-ea8c91a209a3-kube-api-access-s2clt\" (UniqueName: \"kubernetes.io/projected/b44978b4-76f6-44c2-b83d-ea8c91a209a3-kube-api-access-s2clt\") pod \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") "
Nov 02 01:55:05.207105 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.207077    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-xks-ca\") pod \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\" (UID: \"b44978b4-76f6-44c2-b83d-ea8c91a209a3\") "
Nov 02 01:55:05.207254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.207123    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-openshift-ca" pod "b44978b4-76f6-44c2-b83d-ea8c91a209a3" (UID: "b44978b4-76f6-44c2-b83d-ea8c91a209a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:55:05.207841 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.207820    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:05.209266 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.209243    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44978b4-76f6-44c2-b83d-ea8c91a209a3-kube-api-access-s2clt" pod "b44978b4-76f6-44c2-b83d-ea8c91a209a3" (UID: "b44978b4-76f6-44c2-b83d-ea8c91a209a3"). InnerVolumeSpecName "kube-api-access-s2clt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:05.209364 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.209350    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-xks-ca" pod "b44978b4-76f6-44c2-b83d-ea8c91a209a3" (UID: "b44978b4-76f6-44c2-b83d-ea8c91a209a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:55:05.308818 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.308778    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44978b4-76f6-44c2-b83d-ea8c91a209a3-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:05.308818 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.308810    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2clt\" (UniqueName: \"kubernetes.io/projected/b44978b4-76f6-44c2-b83d-ea8c91a209a3-kube-api-access-s2clt\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:05.929898 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.929870    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"
Nov 02 01:55:05.929898 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.929883    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr" event={"ID":"b44978b4-76f6-44c2-b83d-ea8c91a209a3","Type":"ContainerDied","Data":"8fb1b8c8a734c8cf20c6f12e7b57ac5ace2960492256e0576b3f17792d964aee"}
Nov 02 01:55:05.930512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:05.929953    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fb1b8c8a734c8cf20c6f12e7b57ac5ace2960492256e0576b3f17792d964aee"
Nov 02 01:55:06.118048 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:06.118013    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:06.306157 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:06.306024    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:07.766479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:07.766433    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:07.963705 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:07.963670    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:08.152904 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:08.152869    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:08.343792 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:08.343756    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:08.949913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:08.949880    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:09.181132 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.181098    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:09.359835 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.359692    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:09.546982 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.546951    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"]
Nov 02 01:55:09.547809 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.547785    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b44978b4-76f6-44c2-b83d-ea8c91a209a3" containerName="revoke-keys"
Nov 02 01:55:09.547809 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.547808    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44978b4-76f6-44c2-b83d-ea8c91a209a3" containerName="revoke-keys"
Nov 02 01:55:09.548006 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.547991    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44978b4-76f6-44c2-b83d-ea8c91a209a3" containerName="revoke-keys"
Nov 02 01:55:09.551874 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.551847    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:09.657436 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.657402    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-495b48c2\""
Nov 02 01:55:09.657663 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.657643    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-495b48c2-dockercfg-klp48\""
Nov 02 01:55:09.662610 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.662583    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"]
Nov 02 01:55:09.680913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.680886    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"]
Nov 02 01:55:09.686263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.686238    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:09.696079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.695576    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c42d23-3286-4128-aeb7-42bc8e487599-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d\" (UID: \"d9c42d23-3286-4128-aeb7-42bc8e487599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:09.696079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.695707    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klgp\" (UniqueName: \"kubernetes.io/projected/d9c42d23-3286-4128-aeb7-42bc8e487599-kube-api-access-6klgp\") pod \"payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d\" (UID: \"d9c42d23-3286-4128-aeb7-42bc8e487599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:09.763904 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.757314    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"]
Nov 02 01:55:09.799211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.797725    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c42d23-3286-4128-aeb7-42bc8e487599-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d\" (UID: \"d9c42d23-3286-4128-aeb7-42bc8e487599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:09.799211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.797839    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166ad525-f537-4c70-8041-af4f04d0f0dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8\" (UID: \"166ad525-f537-4c70-8041-af4f04d0f0dc\") " pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:09.799211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.797906    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klgp\" (UniqueName: \"kubernetes.io/projected/d9c42d23-3286-4128-aeb7-42bc8e487599-kube-api-access-6klgp\") pod \"payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d\" (UID: \"d9c42d23-3286-4128-aeb7-42bc8e487599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:09.799211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.798127    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4btb\" (UniqueName: \"kubernetes.io/projected/166ad525-f537-4c70-8041-af4f04d0f0dc-kube-api-access-g4btb\") pod \"payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8\" (UID: \"166ad525-f537-4c70-8041-af4f04d0f0dc\") " pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:09.799211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.799124    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c42d23-3286-4128-aeb7-42bc8e487599-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d\" (UID: \"d9c42d23-3286-4128-aeb7-42bc8e487599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:09.871186 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.862808    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klgp\" (UniqueName: \"kubernetes.io/projected/d9c42d23-3286-4128-aeb7-42bc8e487599-kube-api-access-6klgp\") pod \"payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d\" (UID: \"d9c42d23-3286-4128-aeb7-42bc8e487599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:09.872095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.871830    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:09.899119 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.899090    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4btb\" (UniqueName: \"kubernetes.io/projected/166ad525-f537-4c70-8041-af4f04d0f0dc-kube-api-access-g4btb\") pod \"payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8\" (UID: \"166ad525-f537-4c70-8041-af4f04d0f0dc\") " pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:09.899273 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.899223    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166ad525-f537-4c70-8041-af4f04d0f0dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8\" (UID: \"166ad525-f537-4c70-8041-af4f04d0f0dc\") " pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:09.899994 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.899963    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166ad525-f537-4c70-8041-af4f04d0f0dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8\" (UID: \"166ad525-f537-4c70-8041-af4f04d0f0dc\") " pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:09.964081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:09.963782    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4btb\" (UniqueName: \"kubernetes.io/projected/166ad525-f537-4c70-8041-af4f04d0f0dc-kube-api-access-g4btb\") pod \"payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8\" (UID: \"166ad525-f537-4c70-8041-af4f04d0f0dc\") " pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:10.005667 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.005638    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:10.263473 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:10.263448    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c42d23_3286_4128_aeb7_42bc8e487599.slice/crio-d0ced8fcb7c7e4e0be2b60505370c0d80d6b47dce3e848040bbe0228f92a931a WatchSource:0}: Error finding container d0ced8fcb7c7e4e0be2b60505370c0d80d6b47dce3e848040bbe0228f92a931a: Status 404 returned error can't find the container with id d0ced8fcb7c7e4e0be2b60505370c0d80d6b47dce3e848040bbe0228f92a931a
Nov 02 01:55:10.265528 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.265442    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"]
Nov 02 01:55:10.374309 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.374276    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:10.399157 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:10.399114    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166ad525_f537_4c70_8041_af4f04d0f0dc.slice/crio-fdfdd2492fcc41469b4b7c12ee1456ae3b4b017fc941badc20fa8eb700a6a7c1 WatchSource:0}: Error finding container fdfdd2492fcc41469b4b7c12ee1456ae3b4b017fc941badc20fa8eb700a6a7c1: Status 404 returned error can't find the container with id fdfdd2492fcc41469b4b7c12ee1456ae3b4b017fc941badc20fa8eb700a6a7c1
Nov 02 01:55:10.399626 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.399601    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"]
Nov 02 01:55:10.600714 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.600629    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:10.954457 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.954422    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:10.983778 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.983740    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d" event={"ID":"d9c42d23-3286-4128-aeb7-42bc8e487599","Type":"ContainerStarted","Data":"ad0a280c54eb08c62da107ac23e7859b270132cb175dbc8915dd8b07f7de4944"}
Nov 02 01:55:10.984325 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.983802    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d" event={"ID":"d9c42d23-3286-4128-aeb7-42bc8e487599","Type":"ContainerStarted","Data":"d0ced8fcb7c7e4e0be2b60505370c0d80d6b47dce3e848040bbe0228f92a931a"}
Nov 02 01:55:10.984325 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.983825    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:10.985633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.985610    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8" event={"ID":"166ad525-f537-4c70-8041-af4f04d0f0dc","Type":"ContainerStarted","Data":"152691f186fe62cbb7c5cf46fb048bc0484717f88239751252003aff292b055a"}
Nov 02 01:55:10.985756 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.985651    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8" event={"ID":"166ad525-f537-4c70-8041-af4f04d0f0dc","Type":"ContainerStarted","Data":"fdfdd2492fcc41469b4b7c12ee1456ae3b4b017fc941badc20fa8eb700a6a7c1"}
Nov 02 01:55:10.985756 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:10.985700    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:11.105177 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.105129    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d" podStartSLOduration=2.105115667 podStartE2EDuration="2.105115667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:11.104240628 +0000 UTC m=+1404.009705983" watchObservedRunningTime="2026-11-02 01:55:11.105115667 +0000 UTC m=+1404.010581023"
Nov 02 01:55:11.223604 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.223466    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8" podStartSLOduration=2.223447177 podStartE2EDuration="2.223447177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:11.219580845 +0000 UTC m=+1404.125046203" watchObservedRunningTime="2026-11-02 01:55:11.223447177 +0000 UTC m=+1404.128912537"
Nov 02 01:55:11.342305 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.342271    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"]
Nov 02 01:55:11.347200 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.347172    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.400325 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.400182    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:55:11.400598 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.400338    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:55:11.407934 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.407900    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"]
Nov 02 01:55:11.526003 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.525912    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d62c6888-r8kp5\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.526258 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.526239    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76g6\" (UniqueName: \"kubernetes.io/projected/0939ec83-4504-48a0-9fae-2cc7847ebf16-kube-api-access-p76g6\") pod \"maas-api-revoke-sub-d62c6888-r8kp5\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.526335 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.526281    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d62c6888-r8kp5\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.627796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.627766    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76g6\" (UniqueName: \"kubernetes.io/projected/0939ec83-4504-48a0-9fae-2cc7847ebf16-kube-api-access-p76g6\") pod \"maas-api-revoke-sub-d62c6888-r8kp5\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.627962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.627806    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d62c6888-r8kp5\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.627962 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.627866    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d62c6888-r8kp5\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.628715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.628638    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d62c6888-r8kp5\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.630655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.630628    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d62c6888-r8kp5\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.689731 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.689703    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76g6\" (UniqueName: \"kubernetes.io/projected/0939ec83-4504-48a0-9fae-2cc7847ebf16-kube-api-access-p76g6\") pod \"maas-api-revoke-sub-d62c6888-r8kp5\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:11.967810 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:11.967780    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:12.139922 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:55:12.139889    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:55:12.361557 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:12.361529    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0939ec83_4504_48a0_9fae_2cc7847ebf16.slice/crio-c7170322ce558fb5e310f69d1ac1b6eed36ca7bd95b9c29edaefa4fb8c069cbf WatchSource:0}: Error finding container c7170322ce558fb5e310f69d1ac1b6eed36ca7bd95b9c29edaefa4fb8c069cbf: Status 404 returned error can't find the container with id c7170322ce558fb5e310f69d1ac1b6eed36ca7bd95b9c29edaefa4fb8c069cbf
Nov 02 01:55:12.362146 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:12.362127    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"]
Nov 02 01:55:13.009520 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:13.009427    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5" event={"ID":"0939ec83-4504-48a0-9fae-2cc7847ebf16","Type":"ContainerStarted","Data":"f397a7478deb559853894faab5f1d8977f6fa4b24c456a66a258a880b55e48fe"}
Nov 02 01:55:13.009520 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:13.009488    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5" event={"ID":"0939ec83-4504-48a0-9fae-2cc7847ebf16","Type":"ContainerStarted","Data":"c7170322ce558fb5e310f69d1ac1b6eed36ca7bd95b9c29edaefa4fb8c069cbf"}
Nov 02 01:55:13.124694 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:13.124636    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5" podStartSLOduration=2.124620493 podStartE2EDuration="2.124620493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:13.123565931 +0000 UTC m=+1406.029031290" watchObservedRunningTime="2026-11-02 01:55:13.124620493 +0000 UTC m=+1406.030085849"
Nov 02 01:55:14.020105 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.020050    4912 generic.go:356] "Generic (PLEG): container finished" podID="0939ec83-4504-48a0-9fae-2cc7847ebf16" containerID="f397a7478deb559853894faab5f1d8977f6fa4b24c456a66a258a880b55e48fe" exitCode=0
Nov 02 01:55:14.020105 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.020083    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5" event={"ID":"0939ec83-4504-48a0-9fae-2cc7847ebf16","Type":"ContainerDied","Data":"f397a7478deb559853894faab5f1d8977f6fa4b24c456a66a258a880b55e48fe"}
Nov 02 01:55:14.341209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.341133    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:14.564079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.564030    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:14.714239 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.714203    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"]
Nov 02 01:55:14.719074 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.719031    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:14.770071 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.770013    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089666e8-41da-405d-a8d9-11b81fcb699b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7\" (UID: \"089666e8-41da-405d-a8d9-11b81fcb699b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:14.770249 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.770170    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vtff\" (UniqueName: \"kubernetes.io/projected/089666e8-41da-405d-a8d9-11b81fcb699b-kube-api-access-7vtff\") pod \"payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7\" (UID: \"089666e8-41da-405d-a8d9-11b81fcb699b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:14.779362 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.773717    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b9a2fcb6\""
Nov 02 01:55:14.780360 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.780331    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"]
Nov 02 01:55:14.893081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.872601    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vtff\" (UniqueName: \"kubernetes.io/projected/089666e8-41da-405d-a8d9-11b81fcb699b-kube-api-access-7vtff\") pod \"payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7\" (UID: \"089666e8-41da-405d-a8d9-11b81fcb699b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:14.893081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.872980    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089666e8-41da-405d-a8d9-11b81fcb699b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7\" (UID: \"089666e8-41da-405d-a8d9-11b81fcb699b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:14.893081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.873971    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089666e8-41da-405d-a8d9-11b81fcb699b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7\" (UID: \"089666e8-41da-405d-a8d9-11b81fcb699b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:14.933844 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.932264    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:55:14.933844 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.933594    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:55:14.936615 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:14.936588    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vtff\" (UniqueName: \"kubernetes.io/projected/089666e8-41da-405d-a8d9-11b81fcb699b-kube-api-access-7vtff\") pod \"payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7\" (UID: \"089666e8-41da-405d-a8d9-11b81fcb699b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:15.035619 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.035588    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:15.306002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.305937    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:15.388121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.388086    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0939ec83-4504-48a0-9fae-2cc7847ebf16-kube-api-access-p76g6\" (UniqueName: \"kubernetes.io/projected/0939ec83-4504-48a0-9fae-2cc7847ebf16-kube-api-access-p76g6\") pod \"0939ec83-4504-48a0-9fae-2cc7847ebf16\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") "
Nov 02 01:55:15.388121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.388127    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-openshift-ca\") pod \"0939ec83-4504-48a0-9fae-2cc7847ebf16\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") "
Nov 02 01:55:15.388328 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.388208    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-xks-ca\") pod \"0939ec83-4504-48a0-9fae-2cc7847ebf16\" (UID: \"0939ec83-4504-48a0-9fae-2cc7847ebf16\") "
Nov 02 01:55:15.388532 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.388505    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-openshift-ca" pod "0939ec83-4504-48a0-9fae-2cc7847ebf16" (UID: "0939ec83-4504-48a0-9fae-2cc7847ebf16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:55:15.388695 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.388674    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:15.390482 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.390461    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-xks-ca" pod "0939ec83-4504-48a0-9fae-2cc7847ebf16" (UID: "0939ec83-4504-48a0-9fae-2cc7847ebf16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:55:15.390541 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.390514    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0939ec83-4504-48a0-9fae-2cc7847ebf16-kube-api-access-p76g6" pod "0939ec83-4504-48a0-9fae-2cc7847ebf16" (UID: "0939ec83-4504-48a0-9fae-2cc7847ebf16"). InnerVolumeSpecName "kube-api-access-p76g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:15.478822 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.478730    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"]
Nov 02 01:55:15.490544 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.490513    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p76g6\" (UniqueName: \"kubernetes.io/projected/0939ec83-4504-48a0-9fae-2cc7847ebf16-kube-api-access-p76g6\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:15.490660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.490548    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0939ec83-4504-48a0-9fae-2cc7847ebf16-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:15.525095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:15.525070    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:16.043956 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:16.043917    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7" event={"ID":"089666e8-41da-405d-a8d9-11b81fcb699b","Type":"ContainerStarted","Data":"27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859"}
Nov 02 01:55:16.044681 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:16.043985    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:16.044681 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:16.044013    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7" event={"ID":"089666e8-41da-405d-a8d9-11b81fcb699b","Type":"ContainerStarted","Data":"495c5dd57b705f1618199577ba1803384da0e1a28c92fb74b98e82f963f60451"}
Nov 02 01:55:16.045758 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:16.045725    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"
Nov 02 01:55:16.045758 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:16.045744    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5" event={"ID":"0939ec83-4504-48a0-9fae-2cc7847ebf16","Type":"ContainerDied","Data":"c7170322ce558fb5e310f69d1ac1b6eed36ca7bd95b9c29edaefa4fb8c069cbf"}
Nov 02 01:55:16.045895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:16.045785    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7170322ce558fb5e310f69d1ac1b6eed36ca7bd95b9c29edaefa4fb8c069cbf"
Nov 02 01:55:16.159321 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:16.159264    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7" podStartSLOduration=2.159245927 podStartE2EDuration="2.159245927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:16.159097727 +0000 UTC m=+1409.064563080" watchObservedRunningTime="2026-11-02 01:55:16.159245927 +0000 UTC m=+1409.064711287"
Nov 02 01:55:16.980824 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:16.980783    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:17.178787 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:17.178755    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:17.371504 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:17.371471    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:17.556928 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:17.556887    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:17.776172 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:17.776081    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:20.354722 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:20.354686    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:22.000037 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:22.000005    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:55:22.000474 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:22.000297    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:55:22.094599 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:22.094566    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:22.819010 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:22.818974    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:23.750794 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:23.750763    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:23.878818 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:23.878789    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"]
Nov 02 01:55:23.879617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:23.879594    4912 cpu_manager.go:414] "containerMap: removing container" podUID="0939ec83-4504-48a0-9fae-2cc7847ebf16" containerName="revoke-keys"
Nov 02 01:55:23.879617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:23.879614    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0939ec83-4504-48a0-9fae-2cc7847ebf16" containerName="revoke-keys"
Nov 02 01:55:23.879889 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:23.879865    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="0939ec83-4504-48a0-9fae-2cc7847ebf16" containerName="revoke-keys"
Nov 02 01:55:23.884218 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:23.884200    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:23.960490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:23.960442    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:24.003176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:23.998603    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"]
Nov 02 01:55:24.004403 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.004377    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"]
Nov 02 01:55:24.009323 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.009267    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.019319 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.019294    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6kv\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-kube-api-access-2c6kv\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.019464 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.019352    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.019464 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.019407    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.019464 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.019450    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eac713a-c04b-4a08-9a96-3274f245bb9e-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.019646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.019486    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.019646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.019531    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.019646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.019594    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.019821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.019729    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.019882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.019856    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.091900 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.091872    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"]
Nov 02 01:55:24.121683 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.121646    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eac713a-c04b-4a08-9a96-3274f245bb9e-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.121865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.121703    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.121865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.121735    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.121865 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.121774    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.121987 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.121875    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.121987 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.121946    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.122112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122000    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.122112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122090    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.122254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122148    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.122310 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122292    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj6j9\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-kube-api-access-nj6j9\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.122384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122364    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.122443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122406    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.122443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122437    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.123174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122573    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.123174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122609    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6kv\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-kube-api-access-2c6kv\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.123174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122657    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.123174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122705    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.123174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122709    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.123174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122740    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.123174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.122812    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.123612 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.123381    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.123726 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.123683    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.124267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.124249    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eac713a-c04b-4a08-9a96-3274f245bb9e-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.126566 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.126537    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.127100 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.127040    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.183830 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.183798    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.184026 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.184001    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6kv\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-kube-api-access-2c6kv\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.201797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.201771    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:24.223946 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.223921    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj6j9\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-kube-api-access-nj6j9\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.224107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.223961    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.224107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.223982    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.224107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.224007    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.224291 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.224137    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.224291 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.224230    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.224399 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.224303    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.224399 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.224369    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.224510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.224390    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.224510 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.224460    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.228723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.228361    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.228723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.228448    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.228723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.228633    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.228723 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.228650    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.231422 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.231401    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.232195 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.232179    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.301648 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.301579    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.302843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.302801    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj6j9\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-kube-api-access-nj6j9\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.329930 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.329894    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:24.652784 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:24.652752    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eac713a_c04b_4a08_9a96_3274f245bb9e.slice/crio-f5c0841433e4c561ed31a9d5539c71e82b7bbef92ec4a6a15cde1eff8532b18a WatchSource:0}: Error finding container f5c0841433e4c561ed31a9d5539c71e82b7bbef92ec4a6a15cde1eff8532b18a: Status 404 returned error can't find the container with id f5c0841433e4c561ed31a9d5539c71e82b7bbef92ec4a6a15cde1eff8532b18a
Nov 02 01:55:24.654138 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.654109    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"]
Nov 02 01:55:24.655836 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.655800    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:24.655944 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.655885    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:24.655944 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.655932    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:24.768680 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.767843    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"]
Nov 02 01:55:24.772890 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.772847    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:24.773002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.772936    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:24.773002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:24.772975    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:25.145699 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.145658    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" event={"ID":"31c4c212-d4ce-4b23-be07-a85fb4ded4f7","Type":"ContainerStarted","Data":"0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec"}
Nov 02 01:55:25.145907 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.145731    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" event={"ID":"31c4c212-d4ce-4b23-be07-a85fb4ded4f7","Type":"ContainerStarted","Data":"47ebf56561b763e18286ad1570bc2ac7fd1832fb2301cc301a26996be79a8a2c"}
Nov 02 01:55:25.148686 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.148649    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" event={"ID":"5eac713a-c04b-4a08-9a96-3274f245bb9e","Type":"ContainerStarted","Data":"02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c"}
Nov 02 01:55:25.148808 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.148712    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" event={"ID":"5eac713a-c04b-4a08-9a96-3274f245bb9e","Type":"ContainerStarted","Data":"f5c0841433e4c561ed31a9d5539c71e82b7bbef92ec4a6a15cde1eff8532b18a"}
Nov 02 01:55:25.171481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.171443    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:25.202632 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.202600    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:25.204370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.204336    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:25.204770 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.204387    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:25.312613 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.312548    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podStartSLOduration=2.312530373 podStartE2EDuration="2.312530373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:25.265499466 +0000 UTC m=+1418.170964826" watchObservedRunningTime="2026-11-02 01:55:25.312530373 +0000 UTC m=+1418.217995729"
Nov 02 01:55:25.331177 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.331147    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:25.332863 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.332841    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:25.332932 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.332891    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:25.388385 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.388318    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podStartSLOduration=2.388295194 podStartE2EDuration="2.388295194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:25.38448831 +0000 UTC m=+1418.289953667" watchObservedRunningTime="2026-11-02 01:55:25.388295194 +0000 UTC m=+1418.293760553"
Nov 02 01:55:25.520466 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.520371    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"]
Nov 02 01:55:25.525838 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.525812    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.595892 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.595760    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"]
Nov 02 01:55:25.646557 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.646521    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.646769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.646729    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.646850 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.646774    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cht2\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-kube-api-access-8cht2\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.646850 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.646824    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.646992 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.646973    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.647109 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.647085    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f069-354b-490b-b5cf-e002a065f0d6-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.647186 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.647163    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.647339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.647321    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.647411 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.647385    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.771143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.751014    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.771624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.771380    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.771624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.771497    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.771624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.771577    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.771787 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.771743    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.771846 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.771802    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cht2\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-kube-api-access-8cht2\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.771898 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.771879    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.784508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.771946    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.784508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.772035    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f069-354b-490b-b5cf-e002a065f0d6-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.784508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.782462    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.784508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.782709    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.784508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.782973    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.784508 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.783585    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.785611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.785557    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.796192 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.794596    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.796408 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.796378    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f069-354b-490b-b5cf-e002a065f0d6-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.833647 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.833611    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"]
Nov 02 01:55:25.840129 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.840100    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.840721 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.840705    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:25.845875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.845857    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cht2\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-kube-api-access-8cht2\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.848191 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.848166    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"]
Nov 02 01:55:25.866513 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.865306    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:25.978176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.977021    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czb8\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-kube-api-access-6czb8\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:25.978176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.977116    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:25.978176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.977239    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16fb43b-1c92-40fb-935b-a25a511b71d1-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:25.978176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.977279    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:25.978176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.977344    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:25.978176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.977381    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:25.978176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.977423    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:25.978176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.977504    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:25.978176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:25.977579    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.032137 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.030525    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"]
Nov 02 01:55:26.032137 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.030601    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"]
Nov 02 01:55:26.032137 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.030733    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.092850    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16fb43b-1c92-40fb-935b-a25a511b71d1-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.092915    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.092977    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.093013    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.093072    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.093155    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.093227    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.093314    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czb8\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-kube-api-access-6czb8\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.093363    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.096098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.093878    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.120096 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.109357    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.120096 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.109612    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.120096 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.114099    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.124096 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.123530    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16fb43b-1c92-40fb-935b-a25a511b71d1-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.125159 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.125106    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.126369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.126341    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.213494 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.212583    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:26.213494 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.212638    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:26.220099 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.216473    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4633de5-ffcd-4056-8677-cd5426f8a643-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.220468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.220444    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxp8\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-kube-api-access-cnxp8\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.220729 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.220683    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.221209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.220825    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.221209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.220905    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.221209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.221010    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.221901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.221562    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.221901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.221605    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.221901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.221636    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.223573 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.223525    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.233951 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.233208    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czb8\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-kube-api-access-6czb8\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.303707 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.303604    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:26.325558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.325503    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.325768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.325591    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.325768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.325655    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4633de5-ffcd-4056-8677-cd5426f8a643-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.325768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.325729    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxp8\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-kube-api-access-cnxp8\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.325952 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.325811    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.325952 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.325944    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.326079 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.326029    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.326194 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.326176    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.326561 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.326272    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.326968 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.326940    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.328660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.327271    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.328660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.327612    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4633de5-ffcd-4056-8677-cd5426f8a643-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.328660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.327729    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.328660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.328036    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.332170 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.331340    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:26.332170 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.331397    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:26.334587 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.334039    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.335648 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.335214    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.432977 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.432931    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.440850 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.440822    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxp8\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-kube-api-access-cnxp8\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.627693 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.627645    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"]
Nov 02 01:55:26.629695 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:26.629663    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b3f069_354b_490b_b5cf_e002a065f0d6.slice/crio-c9da3e34e326c27da3c3cc7d4489a9f45bfc10b2339a01243e76fac7d8667961 WatchSource:0}: Error finding container c9da3e34e326c27da3c3cc7d4489a9f45bfc10b2339a01243e76fac7d8667961: Status 404 returned error can't find the container with id c9da3e34e326c27da3c3cc7d4489a9f45bfc10b2339a01243e76fac7d8667961
Nov 02 01:55:26.633225 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.632280    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:26.633225 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.632344    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:26.633225 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.632394    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:26.735705 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.732416    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:26.997078 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:26.988565    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"]
Nov 02 01:55:27.124397 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.118264    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:55:27.144689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.144555    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:27.144689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.144641    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:27.144862 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.144705    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:27.207382 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.205938    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:27.207382 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.205993    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:27.330758 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.330728    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:27.330986 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.330957    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:27.332272 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.332224    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" event={"ID":"a16fb43b-1c92-40fb-935b-a25a511b71d1","Type":"ContainerStarted","Data":"f0e4d48cbfcd7afe4f2d8e4f0913bcca0af1735062d2ecf7663dd14fe558887f"}
Nov 02 01:55:27.337151 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.337125    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" event={"ID":"25b3f069-354b-490b-b5cf-e002a065f0d6","Type":"ContainerStarted","Data":"c4160f81f7ff2f1358b867e327aa245568f6a323ec178bee9a3f32c68a399215"}
Nov 02 01:55:27.337430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.337388    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" event={"ID":"25b3f069-354b-490b-b5cf-e002a065f0d6","Type":"ContainerStarted","Data":"c9da3e34e326c27da3c3cc7d4489a9f45bfc10b2339a01243e76fac7d8667961"}
Nov 02 01:55:27.538914 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.538786    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podStartSLOduration=2.538763888 podStartE2EDuration="2.538763888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:27.482557483 +0000 UTC m=+1420.388022843" watchObservedRunningTime="2026-11-02 01:55:27.538763888 +0000 UTC m=+1420.444229248"
Nov 02 01:55:27.641656 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:27.639669    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4633de5_ffcd_4056_8677_cd5426f8a643.slice/crio-bfdb00b4769ae1348b9307c720c464cb30fa40bede2a99e885d23c235f35a092 WatchSource:0}: Error finding container bfdb00b4769ae1348b9307c720c464cb30fa40bede2a99e885d23c235f35a092: Status 404 returned error can't find the container with id bfdb00b4769ae1348b9307c720c464cb30fa40bede2a99e885d23c235f35a092
Nov 02 01:55:27.641656 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.641108    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"]
Nov 02 01:55:27.642386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.642354    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:27.642503 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.642417    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:27.642503 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.642445    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:55:27.895107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.877627    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:27.905092 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.904447    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:27.905092 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:27.904534    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:28.207095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.202286    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:28.207095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.202339    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:28.364843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.346189    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:28.364843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.346257    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:28.568148 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.563809    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" event={"ID":"d4633de5-ffcd-4056-8677-cd5426f8a643","Type":"ContainerStarted","Data":"8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115"}
Nov 02 01:55:28.568148 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.563879    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" event={"ID":"d4633de5-ffcd-4056-8677-cd5426f8a643","Type":"ContainerStarted","Data":"bfdb00b4769ae1348b9307c720c464cb30fa40bede2a99e885d23c235f35a092"}
Nov 02 01:55:28.606543 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.606504    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" event={"ID":"a16fb43b-1c92-40fb-935b-a25a511b71d1","Type":"ContainerStarted","Data":"b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58"}
Nov 02 01:55:28.745116 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.742184    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podStartSLOduration=3.742161824 podStartE2EDuration="3.742161824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:28.713542086 +0000 UTC m=+1421.619007447" watchObservedRunningTime="2026-11-02 01:55:28.742161824 +0000 UTC m=+1421.647627201"
Nov 02 01:55:28.784166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.784130    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:28.791196 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.791161    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:28.791493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.791463    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:28.801646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.801611    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:28.870146 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.866583    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podStartSLOduration=3.86656275 podStartE2EDuration="3.86656275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:28.847693852 +0000 UTC m=+1421.753159212" watchObservedRunningTime="2026-11-02 01:55:28.86656275 +0000 UTC m=+1421.772028108"
Nov 02 01:55:28.870732 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.870703    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:28.870918 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:28.870895    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:29.049088 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.042113    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.208140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.207680    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.208140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.207763    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:29.279084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.277342    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.376091 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.374193    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.376091 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.374269    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:29.376091 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.374361    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:29.377134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.376697    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.377134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.376768    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:29.447227 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.444467    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.535144 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.531200    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.759085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.736104    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.759514 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.759474    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:29.788976 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.788480    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.840376 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.840326    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.852179 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.840652    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.852512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.840741    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.852652 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.847335    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:29.892342 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.887280    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.892342 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.887675    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.892342 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.891707    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.892342 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.891751    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:29.991755 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:29.967029    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.052463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.052368    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.063185 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.052700    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 01:55:30.063185 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.057143    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.063650 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.063627    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:55:30.349412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.308529    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.349412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.308588    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:30.377760 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.355255    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.377760 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.355324    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:30.413429 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.413353    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.469207 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.469157    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.469859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.469834    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:55:30.665934 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.665892    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:55:30.666860 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.666836    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:55:30.742095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.739475    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.742095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.739546    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:30.742095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.741194    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.757529 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.748950    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.769243 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.769177    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:30.794484 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.794440    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.819226 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.819190    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:55:30.943145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.925571    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 01:55:30.943145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.940796    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:55:30.943145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.941832    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.947333 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.947303    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:55:30.968100 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.967333    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.968100 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.967401    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:30.968100 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.967498    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 01:55:30.968100 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.968035    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:55:30.997123 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.968964    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Nov 02 01:55:30.997123 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.969218    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:55:30.997123 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.969379    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 01:55:30.997123 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.969568    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:55:30.997123 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.969614    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:55:30.997123 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:30.969651    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:55:31.033626 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.032849    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:31.068645 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.067150    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:31.068645 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.067482    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 01:55:31.068645 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.067734    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:55:31.068645 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.067791    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:55:31.068645 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.068436    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 01:55:31.267960 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.267867    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.268246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.268215    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:31.357493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.357435    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.358173 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.358125    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:31.392358 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.392228    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.392358 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.392289    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:31.472670 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.470439    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 02 01:55:31.538679 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.538561    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:31.712297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.712240    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:31.812830 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.812731    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.813130 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.813100    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:31.843292 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.825161    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 01:55:31.904087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.902206    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.904087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.902275    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:31.959028 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.958962    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 02 01:55:32.098375 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.983145    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 01:55:32.122134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.983195    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 01:55:32.184111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.983214    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:55:32.184111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:31.983232    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 01:55:32.184111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.011138    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:55:32.184111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.067457    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.184111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.069071    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.184111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.069104    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 01:55:32.249087 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.224218    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.265179 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.265130    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:32.363174 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.361451    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.443710 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.443671    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.443959 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.443929    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:32.450933 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.449330    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.483143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.474200    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.587085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.511681    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.587085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.511740    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:32.731109 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.713158    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.776097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.775922    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.776097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.775986    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:32.837099 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.826312    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.938278 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.938113    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.938278 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:32.938187    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:33.082712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.082609    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.106449 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.106405    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.106994 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.106963    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.178448 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.178397    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.208085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.182525    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Nov 02 01:55:33.257114 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.192017    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 01:55:33.257114 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.202186    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:33.257114 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.251763    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:33.257114 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.212803    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Nov 02 01:55:33.257114 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.212828    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.257114 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.212770    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 01:55:33.257114 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.212846    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.443383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.435459    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:33.443383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.435529    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:33.443383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.436846    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:33.443383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.436890    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:33.467424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.448854    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.530086 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.501859    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.857461 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.843655    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:33.887032 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.886980    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.992085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:33.966674    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.091085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.089043    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.091085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.089144    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:34.132388 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.124253    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.275757 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.150953    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.276034 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.275998    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:34.276224 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.258360    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:34.276784 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.159286    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:34.277472 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.176690    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:34.277642 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.277621    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:34.307124 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.295670    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.385338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.376248    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.385338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.376319    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:34.755149 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.742415    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.786691 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.786645    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": context deadline exceeded"
Nov 02 01:55:34.793197 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.793155    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.795459 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.794370    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.822446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.795666    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.822446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.810166    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.822446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.810460    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.923121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.903110    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.923121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.903203    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:34.923121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.903330    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.923121 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:34.903366    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:35.033141 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.020275    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.097926 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.097886    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.139241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.139196    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:35.141495 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.140829    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.265777 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.265737    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.266127 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.266091    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:35.355143 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.332020    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.357366 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.357321    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.357617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.357584    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:35.381205 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.381157    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:35.426120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.426075    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.428269 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:55:35.382719    4912 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.112s"
Nov 02 01:55:35.532084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.519356    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:35.533318 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.384108    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.557103 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.382119    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.637362 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.620736    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.653100 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.653035    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:35.680008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.679975    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:55:35.776910 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.776827    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:35.821380 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.821347    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.821795 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.821749    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:35.850493 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.850446    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.851449 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.851413    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.864335 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.851918    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.864957 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.864920    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.865503 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.865465    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.898436 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.898341    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.898707 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:35.898663    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:36.190144 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.181942    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Nov 02 01:55:36.190144 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.183080    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:55:36.213553 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.213520    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 01:55:36.213848 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.213814    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 01:55:36.303816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.303754    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.314259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.314211    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:36.314531 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.314506    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:36.357598 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.357567    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 01:55:36.358021 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.357978    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 01:55:36.426514 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.426456    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.550332 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.550229    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.571124 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.571084    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.571443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.571405    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 01:55:36.573523 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.573495    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": context deadline exceeded"
Nov 02 01:55:36.742243 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.741123    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.762544 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.762508    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:36.762815 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.762785    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:36.863420 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.863313    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.891588 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.866824    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.894216 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.894179    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Nov 02 01:55:36.895536 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.895480    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 01:55:36.909549 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.909515    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 01:55:36.911035 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:36.911007    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 01:55:37.010543 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.004608    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:37.010543 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.004669    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:37.191236 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.191189    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:37.192037 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.191573    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Nov 02 01:55:37.409737 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.408646    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:37.409737 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.408709    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:37.438091 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.437424    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 01:55:37.562422 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.562316    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:37.575088 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.572172    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 01:55:37.748540 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.748492    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:37.764166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.763369    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:37.764166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.763608    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:37.764166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.764100    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:37.867399 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.867343    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 02 01:55:37.869197 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.869171    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:37.869451 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.869411    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:37.897273 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.897216    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:55:37.897641 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.897617    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:37.898807 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.898778    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Nov 02 01:55:37.911030 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.910344    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:37.911322 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:37.911292    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.209136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.205448    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Nov 02 01:55:38.209136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.205872    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:55:38.247727 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.247689    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:38.248002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.247976    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.308449 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.308417    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:38.308771 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.308725    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:38.455084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.431113    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:38.455084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.431221    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.457430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.457388    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 01:55:38.565589 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.565481    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.582259 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.582213    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.584190 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.584161    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:38.586464 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.586444    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:38.616438 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.616407    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 01:55:38.755852 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.755813    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:38.756167 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.756136    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:38.805084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.777389    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 01:55:38.909338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.897196    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.909338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.904454    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.909338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.904761    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 01:55:38.931230 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.931185    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.966126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.935336    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.966126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.946820    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Nov 02 01:55:38.966943 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.966789    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:38.966943 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.966864    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:38.989992 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:38.989891    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.293740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.272760    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Nov 02 01:55:39.293740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.273136    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.307760 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.307721    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.319186 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.319155    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:39.319424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.319403    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:39.384262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.384231    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:39.439183 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.439144    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:39.439474 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.439442    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.449536 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.449514    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 01:55:39.461211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.461153    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 01:55:39.583432 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.583323    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.697088 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.666681    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 01:55:39.744921 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.744890    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 01:55:39.763082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.746490    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 01:55:39.764292 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.748827    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:39.764514 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.764487    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:39.765392 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.765372    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 01:55:39.785348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.785313    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.873729 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.873696    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 01:55:39.874227 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.874002    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 01:55:39.898237 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.898195    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.951489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.905191    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.951489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.905548    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:55:39.951489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.932853    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.951489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.940160    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 01:55:39.951489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:39.951156    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:40.312549 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.312427    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 01:55:40.312549 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.312485    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 01:55:40.336267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.336180    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:40.510550 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.510424    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 01:55:40.666082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.659156    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:40.666624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.666589    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:40.753115 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.753086    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 01:55:40.764146 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.763393    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:55:40.766284 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.764357    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:55:40.767728 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.767704    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:40.938577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.829372    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:40.938577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.878139    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:40.938577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.880440    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:40.938577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.900681    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:40.938577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.913037    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:40.938577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:40.913104    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:41.093330 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.093186    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:41.096416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.095544    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:41.107951 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.106583    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 01:55:41.123614 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.123586    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 01:55:41.142179 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.142134    4912 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Nov 02 01:55:41.176077 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.172171    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 01:55:41.192888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.192861    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 01:55:41.196558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.196018    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 01:55:41.213263 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.213217    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:41.310618 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.310590    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:41.323137 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.319352    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 01:55:41.767097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:41.752886    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:42.073299 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.073210    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:42.073563 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.073532    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:42.314913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.314875    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:42.315611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.315575    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:42.542724 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:55:42.542550    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:55:42.726473 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.726445    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:42.746084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.744378    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 01:55:42.771082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.761154    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:42.771082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.761221    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:42.806134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.805460    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"]
Nov 02 01:55:42.818083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.816419    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:42.927076 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.899291    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-15487453-serving-cert\""
Nov 02 01:55:42.935532 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.934383    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllx2\" (UniqueName: \"kubernetes.io/projected/903ce585-e857-4264-a82c-af2affadc6e3-kube-api-access-vllx2\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:42.935532 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.934632    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-metrics-certs\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:42.935532 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.934679    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/903ce585-e857-4264-a82c-af2affadc6e3-openshift-service-ca\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:42.935532 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.934795    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:42.936262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:42.936187    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"]
Nov 02 01:55:43.064317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.054511    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-metrics-certs\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.064317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.054577    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/903ce585-e857-4264-a82c-af2affadc6e3-openshift-service-ca\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.064317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.054695    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.064317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.054773    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllx2\" (UniqueName: \"kubernetes.io/projected/903ce585-e857-4264-a82c-af2affadc6e3-kube-api-access-vllx2\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.064317 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:55:43.055297    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-15487453-serving-cert: secret "maas-api-e2e-ait-15487453-serving-cert" not found
Nov 02 01:55:43.064317 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:55:43.055365    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls podName:903ce585-e857-4264-a82c-af2affadc6e3 nodeName:}" failed. No retries permitted until 2026-11-02 01:55:43.555345096 +0000 UTC m=+1436.460810452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls") pod "maas-api-e2e-ait-15487453-75d7b55876-5dsnq" (UID: "903ce585-e857-4264-a82c-af2affadc6e3") : secret "maas-api-e2e-ait-15487453-serving-cert" not found
Nov 02 01:55:43.064317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.055968    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/903ce585-e857-4264-a82c-af2affadc6e3-openshift-service-ca\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.064317 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.057252    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:43.132860 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.129268    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-metrics-certs\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.309896 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.309819    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:43.321708 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.312855    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllx2\" (UniqueName: \"kubernetes.io/projected/903ce585-e857-4264-a82c-af2affadc6e3-kube-api-access-vllx2\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.357112 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.343229    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:43.402770 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.402360    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:43.507633 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.507598    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:43.573133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.570477    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:43.581290 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.581250    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.595385 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.595352    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 01:55:43.595922 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.595895    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls\") pod \"maas-api-e2e-ait-15487453-75d7b55876-5dsnq\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.803262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.803221    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:43.803498 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.803307    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:43.803768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.803740    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:43.998007 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:43.997968    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:44.001028 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:44.000997    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 01:55:44.268490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:44.268454    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"]
Nov 02 01:55:44.293687 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:44.291797    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903ce585_e857_4264_a82c_af2affadc6e3.slice/crio-b1a32929f7b8f9ec16ffa2aab8395f68d23e4c4631d13aa48ab11c00aba8ff2c WatchSource:0}: Error finding container b1a32929f7b8f9ec16ffa2aab8395f68d23e4c4631d13aa48ab11c00aba8ff2c: Status 404 returned error can't find the container with id b1a32929f7b8f9ec16ffa2aab8395f68d23e4c4631d13aa48ab11c00aba8ff2c
Nov 02 01:55:45.128257 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:45.128225    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq" event={"ID":"903ce585-e857-4264-a82c-af2affadc6e3","Type":"ContainerStarted","Data":"b1a32929f7b8f9ec16ffa2aab8395f68d23e4c4631d13aa48ab11c00aba8ff2c"}
Nov 02 01:55:45.780192 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:45.780150    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:46.145233 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:46.145190    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq" event={"ID":"903ce585-e857-4264-a82c-af2affadc6e3","Type":"ContainerStarted","Data":"afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66"}
Nov 02 01:55:46.145749 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:46.145300    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:46.265972 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:46.265901    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq" podStartSLOduration=3.474783553 podStartE2EDuration="4.265885848s" totalImagesPullingTime="791.102295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:46.261483708 +0000 UTC m=+1439.166949066" watchObservedRunningTime="2026-11-02 01:55:46.265885848 +0000 UTC m=+1439.171351238"
Nov 02 01:55:47.545709 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.545676    4912 scope.go:122] "RemoveContainer" containerID="8a4a214fca5f119220af4fc6de0334fb92a0a89ff0fc40bd122f5b8f678facf1"
Nov 02 01:55:47.557821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.557797    4912 scope.go:122] "RemoveContainer" containerID="bd86258dd245d92185ec4d3532d7b794910a2b6ad37143aa564a132af9fc177f"
Nov 02 01:55:47.573307 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.573283    4912 scope.go:122] "RemoveContainer" containerID="ab163fb0c6552b4d16ba6dbaa5a0e835e9e4cfeceba11289c509441c08572f77"
Nov 02 01:55:47.586179 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.586158    4912 scope.go:122] "RemoveContainer" containerID="03ca3c0d777bc7c40a3dd0a6e99d6b9353f068dfad2bdc21b3a9dc451bdade6c"
Nov 02 01:55:47.596983 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.596959    4912 scope.go:122] "RemoveContainer" containerID="a40e52e44340d87d6002b9cdec22f98a9e921457a7dc73652ae54d5cc126fcac"
Nov 02 01:55:47.607181 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.607156    4912 scope.go:122] "RemoveContainer" containerID="947029259c3c754ae99cecfd89f9b7f2ae59603c0666c9075ff508c777d702a1"
Nov 02 01:55:47.617443 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.617415    4912 scope.go:122] "RemoveContainer" containerID="b88c82e372332c5c39d57f15678f5d1640780fef8d229ee44d2d4540295f5f60"
Nov 02 01:55:47.628421 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.628394    4912 scope.go:122] "RemoveContainer" containerID="c3a0889d289b6429e25c83fb9c0f1507e6486adc52d8282d59327601bb5a3e5b"
Nov 02 01:55:47.638658 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.638635    4912 scope.go:122] "RemoveContainer" containerID="cd46713b5070073419febaae555bd1d0a540bb1a1db93775b29b83f99884a0d6"
Nov 02 01:55:47.650246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:47.650222    4912 scope.go:122] "RemoveContainer" containerID="ca03bc1a816d218354aefe8ae4155af07234cd50bfc48096c4641b249165058a"
Nov 02 01:55:48.237778 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:48.237747    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:52.163859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:52.163831    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:55:52.966523 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:52.966493    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"]
Nov 02 01:55:52.971285 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:52.971265    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.023369 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.023343    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-495b48c2-serving-cert\""
Nov 02 01:55:53.039224 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.039191    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-metrics-certs\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.039394 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.039324    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.039464 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.039393    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c2mp\" (UniqueName: \"kubernetes.io/projected/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-kube-api-access-8c2mp\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.039591 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.039575    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-openshift-service-ca\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.082042 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.082005    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"]
Nov 02 01:55:53.141383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.141355    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-metrics-certs\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.141571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.141438    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.141571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.141475    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c2mp\" (UniqueName: \"kubernetes.io/projected/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-kube-api-access-8c2mp\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.141571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.141571    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-openshift-service-ca\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.141781 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:55:53.141589    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-serving-cert: secret "maas-api-e2e-ait-495b48c2-serving-cert" not found
Nov 02 01:55:53.141781 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:55:53.141670    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls podName:f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae nodeName:}" failed. No retries permitted until 2026-11-02 01:55:53.64164762 +0000 UTC m=+1446.547112969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls") pod "maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k" (UID: "f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae") : secret "maas-api-e2e-ait-495b48c2-serving-cert" not found
Nov 02 01:55:53.142325 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.142307    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-openshift-service-ca\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.144134 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.144118    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-metrics-certs\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.198992 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.198957    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:53.205548 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.205524    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c2mp\" (UniqueName: \"kubernetes.io/projected/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-kube-api-access-8c2mp\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.374681 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.374647    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:53.563993 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.563948    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:55:53.651393 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.651298    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.654135 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.654107    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls\") pod \"maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:53.887942 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:53.887904    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:54.281940 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:54.281862    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"]
Nov 02 01:55:54.286455 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:55:54.286425    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf709d20a_c6bb_44eb_aa3a_4f0d1c0b80ae.slice/crio-da8c32c05a8a80fa190f73044099f39fe0a3b8aafd89a025fafa247254f781e8 WatchSource:0}: Error finding container da8c32c05a8a80fa190f73044099f39fe0a3b8aafd89a025fafa247254f781e8: Status 404 returned error can't find the container with id da8c32c05a8a80fa190f73044099f39fe0a3b8aafd89a025fafa247254f781e8
Nov 02 01:55:55.247491 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:55.247460    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k" event={"ID":"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae","Type":"ContainerStarted","Data":"8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1"}
Nov 02 01:55:55.247673 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:55.247529    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:55:55.247673 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:55.247556    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k" event={"ID":"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae","Type":"ContainerStarted","Data":"da8c32c05a8a80fa190f73044099f39fe0a3b8aafd89a025fafa247254f781e8"}
Nov 02 01:55:55.378185 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:55.378046    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k" podStartSLOduration=2.54003057 podStartE2EDuration="3.378030934s" totalImagesPullingTime="838.000364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:55.373599969 +0000 UTC m=+1448.279065327" watchObservedRunningTime="2026-11-02 01:55:55.378030934 +0000 UTC m=+1448.283496291"
Nov 02 01:55:58.245736 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:55:58.245698    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:00.453625 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.453585    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"]
Nov 02 01:56:00.459907 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.459874    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.560186 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.560152    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"]
Nov 02 01:56:00.567043 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.567001    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.567277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.567111    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.567277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.567173    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.567277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.567200    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.567277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.567235    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4888s\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-kube-api-access-4888s\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.567512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.567455    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.567630 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.567608    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.567693 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.567667    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.567693 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.567689    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d88d562-14a0-4479-97bc-e9dddfcee365-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.668917 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.668879    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.669107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.668968    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.669107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.668998    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.669107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.669019    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d88d562-14a0-4479-97bc-e9dddfcee365-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.669107 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.669086    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.669303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.669131    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.669303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.669191    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.669303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.669218    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.669303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.669243    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4888s\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-kube-api-access-4888s\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.670093 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.669526    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.671888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.670258    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.671888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.670313    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d88d562-14a0-4479-97bc-e9dddfcee365-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.671888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.670336    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.671888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.670431    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.672161 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.672104    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.672465 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.672440    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.730591 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.730512    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.730591 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.730580    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4888s\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-kube-api-access-4888s\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xmtdb\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:00.776661 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:00.776629    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:01.225177 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:56:01.225148    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d88d562_14a0_4479_97bc_e9dddfcee365.slice/crio-c07ab43c12af6519b477844e0382331f668ecbb0b40aeea4c2c0231ba8665c69 WatchSource:0}: Error finding container c07ab43c12af6519b477844e0382331f668ecbb0b40aeea4c2c0231ba8665c69: Status 404 returned error can't find the container with id c07ab43c12af6519b477844e0382331f668ecbb0b40aeea4c2c0231ba8665c69
Nov 02 01:56:01.228139 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:01.228098    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:56:01.228230 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:01.228186    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:56:01.228230 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:01.228222    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:56:01.231349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:01.231327    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"]
Nov 02 01:56:01.347082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:01.340516    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:56:01.397123 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:01.393309    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" event={"ID":"5d88d562-14a0-4479-97bc-e9dddfcee365","Type":"ContainerStarted","Data":"c07ab43c12af6519b477844e0382331f668ecbb0b40aeea4c2c0231ba8665c69"}
Nov 02 01:56:02.407914 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:02.407882    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" event={"ID":"5d88d562-14a0-4479-97bc-e9dddfcee365","Type":"ContainerStarted","Data":"e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0"}
Nov 02 01:56:02.591419 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:02.591337    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" podStartSLOduration=2.591322172 podStartE2EDuration="2.591322172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:02.585289512 +0000 UTC m=+1455.490754874" watchObservedRunningTime="2026-11-02 01:56:02.591322172 +0000 UTC m=+1455.496787529"
Nov 02 01:56:02.777556 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:02.777463    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:02.789338 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:02.789299    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xmtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 01:56:02.789681 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:02.789648    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 01:56:03.213198 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.213163    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"]
Nov 02 01:56:03.225738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.225147    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.291223 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.291189    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b9a2fcb6-serving-cert\""
Nov 02 01:56:03.317083 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.311098    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"]
Nov 02 01:56:03.335440 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.335396    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95b99c5-0fbc-4c50-8611-20e3cbb665df-openshift-service-ca\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.335760 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.335740    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.336674 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.336651    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbw8\" (UniqueName: \"kubernetes.io/projected/c95b99c5-0fbc-4c50-8611-20e3cbb665df-kube-api-access-9pbw8\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.336968 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.336950    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-metrics-certs\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.438797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.438758    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbw8\" (UniqueName: \"kubernetes.io/projected/c95b99c5-0fbc-4c50-8611-20e3cbb665df-kube-api-access-9pbw8\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.439301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.438888    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-metrics-certs\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.439301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.439136    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95b99c5-0fbc-4c50-8611-20e3cbb665df-openshift-service-ca\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.439301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.439171    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.439463 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:56:03.439362    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-serving-cert: secret "maas-api-e2e-migrate-b9a2fcb6-serving-cert" not found
Nov 02 01:56:03.439463 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:56:03.439439    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls podName:c95b99c5-0fbc-4c50-8611-20e3cbb665df nodeName:}" failed. No retries permitted until 2026-11-02 01:56:03.939416586 +0000 UTC m=+1456.844881924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls") pod "maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn" (UID: "c95b99c5-0fbc-4c50-8611-20e3cbb665df") : secret "maas-api-e2e-migrate-b9a2fcb6-serving-cert" not found
Nov 02 01:56:03.440408 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.440383    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95b99c5-0fbc-4c50-8611-20e3cbb665df-openshift-service-ca\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.442738 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.442707    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-metrics-certs\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.446248 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.446224    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:03.504514 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.504433    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbw8\" (UniqueName: \"kubernetes.io/projected/c95b99c5-0fbc-4c50-8611-20e3cbb665df-kube-api-access-9pbw8\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.690875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.690839    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:03.777682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.777585    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xmtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 01:56:03.778501 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.777911    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 01:56:03.881582 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.881547    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:03.954097 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.949712    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:03.978089 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:03.977749    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls\") pod \"maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:04.144234 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:04.144200    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:04.586469 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:04.586420    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"]
Nov 02 01:56:04.588938 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:56:04.588914    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95b99c5_0fbc_4c50_8611_20e3cbb665df.slice/crio-1994459f802bde1a655dcbaada9ddac523951e3d8d6537b809054d26ca494158 WatchSource:0}: Error finding container 1994459f802bde1a655dcbaada9ddac523951e3d8d6537b809054d26ca494158: Status 404 returned error can't find the container with id 1994459f802bde1a655dcbaada9ddac523951e3d8d6537b809054d26ca494158
Nov 02 01:56:04.777608 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:04.777561    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xmtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 01:56:04.777795 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:04.777620    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 01:56:05.504004 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:05.503961    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn" event={"ID":"c95b99c5-0fbc-4c50-8611-20e3cbb665df","Type":"ContainerStarted","Data":"1994459f802bde1a655dcbaada9ddac523951e3d8d6537b809054d26ca494158"}
Nov 02 01:56:05.778538 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:05.778446    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xmtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 01:56:05.779126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:05.779092    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 01:56:06.521136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:06.521048    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn" event={"ID":"c95b99c5-0fbc-4c50-8611-20e3cbb665df","Type":"ContainerStarted","Data":"cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81"}
Nov 02 01:56:06.521374 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:06.521355    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:06.648022 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:06.647964    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn" podStartSLOduration=2.797924633 podStartE2EDuration="3.647950461s" totalImagesPullingTime="850.025828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:06.645271801 +0000 UTC m=+1459.550737158" watchObservedRunningTime="2026-11-02 01:56:06.647950461 +0000 UTC m=+1459.553415817"
Nov 02 01:56:06.792182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:06.789551    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xmtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 01:56:06.792182 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:06.789611    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 01:56:07.777957 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:07.777908    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xmtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 01:56:07.778316 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:07.777967    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 01:56:08.355727 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:08.355686    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:08.777306 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:08.777237    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xmtdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 01:56:08.777526 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:08.777332    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 01:56:10.196898 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:10.194722    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:10.196898 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:10.196116    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:10.197521 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:10.197498    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 01:56:12.156246 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:56:12.156195    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:56:12.535857 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:12.535780    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:56:13.278154 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:13.278117    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:18.185741 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:18.185697    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:22.927702 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:22.927662    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:27.679407 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:27.679328    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:30.605687 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:30.605640    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:32.410158 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:32.410123    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:37.510197 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:37.510162    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:42.178279 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:56:42.178242    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:56:42.779843 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:42.779805    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:46.133624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:46.133574    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:47.607765 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:47.607738    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:56:47.615512 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:47.615492    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:56:47.662913 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:47.662349    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 01:56:47.685389 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:47.685364    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 01:56:47.835325 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:47.835288    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:48.230272 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:48.230238    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:48.401176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:48.401142    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:52.972638 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:52.972599    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:54.954728 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:54.954686    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:56:59.294013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:56:59.293975    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:01.655385 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:01.655350    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"]
Nov 02 01:57:01.769126 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:01.769098    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"]
Nov 02 01:57:02.054211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:02.054112    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:02.279656 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:02.279617    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:03.525732 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:03.525692    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:04.533885 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:04.533846    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:06.656376 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:06.656329    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m" podUID="2cbbb0cf-795e-4489-a161-f2002e56c3f7" containerName="payload-processing" containerID="cri-o://b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c" gracePeriod=25
Nov 02 01:57:06.770383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:06.770318    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk" podUID="e117c74a-26e9-49a4-9429-45b68fd1a634" containerName="payload-pre-processing" containerID="cri-o://18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743" gracePeriod=25
Nov 02 01:57:06.978183 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:06.978076    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:57:07.011689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.011658    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbbb0cf-795e-4489-a161-f2002e56c3f7-kube-api-access-xrlhp\" (UniqueName: \"kubernetes.io/projected/2cbbb0cf-795e-4489-a161-f2002e56c3f7-kube-api-access-xrlhp\") pod \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\" (UID: \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\") "
Nov 02 01:57:07.012040 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.012006    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbbb0cf-795e-4489-a161-f2002e56c3f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cbbb0cf-795e-4489-a161-f2002e56c3f7-plugins-config-volume\") pod \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\" (UID: \"2cbbb0cf-795e-4489-a161-f2002e56c3f7\") "
Nov 02 01:57:07.013109 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.013050    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbbb0cf-795e-4489-a161-f2002e56c3f7-plugins-config-volume" pod "2cbbb0cf-795e-4489-a161-f2002e56c3f7" (UID: "2cbbb0cf-795e-4489-a161-f2002e56c3f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:07.015122 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.015096    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbbb0cf-795e-4489-a161-f2002e56c3f7-kube-api-access-xrlhp" pod "2cbbb0cf-795e-4489-a161-f2002e56c3f7" (UID: "2cbbb0cf-795e-4489-a161-f2002e56c3f7"). InnerVolumeSpecName "kube-api-access-xrlhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:07.095413 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.095386    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:57:07.117905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.117869    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrlhp\" (UniqueName: \"kubernetes.io/projected/2cbbb0cf-795e-4489-a161-f2002e56c3f7-kube-api-access-xrlhp\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:07.117905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.117909    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cbbb0cf-795e-4489-a161-f2002e56c3f7-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:07.212275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.212191    4912 generic.go:356] "Generic (PLEG): container finished" podID="2cbbb0cf-795e-4489-a161-f2002e56c3f7" containerID="b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c" exitCode=0
Nov 02 01:57:07.212275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.212246    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"
Nov 02 01:57:07.212496 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.212272    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m" event={"ID":"2cbbb0cf-795e-4489-a161-f2002e56c3f7","Type":"ContainerDied","Data":"b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c"}
Nov 02 01:57:07.212496 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.212332    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m" event={"ID":"2cbbb0cf-795e-4489-a161-f2002e56c3f7","Type":"ContainerDied","Data":"c70b0ddce208404544d62b118a26066a922663d4925fa37985fa4c5d0ded18c2"}
Nov 02 01:57:07.212496 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.212348    4912 scope.go:122] "RemoveContainer" containerID="b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c"
Nov 02 01:57:07.214145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.214107    4912 generic.go:356] "Generic (PLEG): container finished" podID="e117c74a-26e9-49a4-9429-45b68fd1a634" containerID="18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743" exitCode=0
Nov 02 01:57:07.214145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.214136    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk" event={"ID":"e117c74a-26e9-49a4-9429-45b68fd1a634","Type":"ContainerDied","Data":"18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743"}
Nov 02 01:57:07.214340 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.214185    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk" event={"ID":"e117c74a-26e9-49a4-9429-45b68fd1a634","Type":"ContainerDied","Data":"356bb3adc8c3421e247ef7273b7acc0de720b46aa6d6f3e1a46b6eb390174437"}
Nov 02 01:57:07.214340 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.214283    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"
Nov 02 01:57:07.221337 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.221316    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e117c74a-26e9-49a4-9429-45b68fd1a634-kube-api-access-wr62q\" (UniqueName: \"kubernetes.io/projected/e117c74a-26e9-49a4-9429-45b68fd1a634-kube-api-access-wr62q\") pod \"e117c74a-26e9-49a4-9429-45b68fd1a634\" (UID: \"e117c74a-26e9-49a4-9429-45b68fd1a634\") "
Nov 02 01:57:07.221523 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.221376    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e117c74a-26e9-49a4-9429-45b68fd1a634-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e117c74a-26e9-49a4-9429-45b68fd1a634-plugins-config-volume\") pod \"e117c74a-26e9-49a4-9429-45b68fd1a634\" (UID: \"e117c74a-26e9-49a4-9429-45b68fd1a634\") "
Nov 02 01:57:07.222271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.222222    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e117c74a-26e9-49a4-9429-45b68fd1a634-plugins-config-volume" pod "e117c74a-26e9-49a4-9429-45b68fd1a634" (UID: "e117c74a-26e9-49a4-9429-45b68fd1a634"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:07.226230 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.226209    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e117c74a-26e9-49a4-9429-45b68fd1a634-kube-api-access-wr62q" pod "e117c74a-26e9-49a4-9429-45b68fd1a634" (UID: "e117c74a-26e9-49a4-9429-45b68fd1a634"). InnerVolumeSpecName "kube-api-access-wr62q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:07.228111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.228088    4912 scope.go:122] "RemoveContainer" containerID="b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c"
Nov 02 01:57:07.228482 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:57:07.228459    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c\": container with ID starting with b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c not found: ID does not exist" containerID="b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c"
Nov 02 01:57:07.228578 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.228490    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c"} err="failed to get container status \"b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c\": rpc error: code = NotFound desc = could not find container \"b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c\": container with ID starting with b8f107bd21e67f1003a14ea741a6ae3a20e515061635c3209ce8d8bdc6c9cc7c not found: ID does not exist"
Nov 02 01:57:07.228578 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.228511    4912 scope.go:122] "RemoveContainer" containerID="18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743"
Nov 02 01:57:07.244590 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.244011    4912 scope.go:122] "RemoveContainer" containerID="18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743"
Nov 02 01:57:07.249002 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:57:07.247314    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743\": container with ID starting with 18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743 not found: ID does not exist" containerID="18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743"
Nov 02 01:57:07.249002 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.247351    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743"} err="failed to get container status \"18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743\": rpc error: code = NotFound desc = could not find container \"18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743\": container with ID starting with 18628667a0ba2cfcfaf0624993a41c2424026203c565a6568a3c0c28f1d10743 not found: ID does not exist"
Nov 02 01:57:07.323318 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.323291    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr62q\" (UniqueName: \"kubernetes.io/projected/e117c74a-26e9-49a4-9429-45b68fd1a634-kube-api-access-wr62q\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:07.323318 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.323314    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e117c74a-26e9-49a4-9429-45b68fd1a634-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:07.434300 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.434269    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"]
Nov 02 01:57:07.439416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.439391    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15487453-5487bb55d5-jrr8m"]
Nov 02 01:57:07.636184 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.636152    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"]
Nov 02 01:57:07.641240 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:07.641217    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15487453-5c75fb8967-kb2rk"]
Nov 02 01:57:09.228525 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:09.228480    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbbb0cf-795e-4489-a161-f2002e56c3f7" path="/var/lib/kubelet/pods/2cbbb0cf-795e-4489-a161-f2002e56c3f7/volumes"
Nov 02 01:57:09.229115 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:09.229096    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e117c74a-26e9-49a4-9429-45b68fd1a634" path="/var/lib/kubelet/pods/e117c74a-26e9-49a4-9429-45b68fd1a634/volumes"
Nov 02 01:57:09.370464 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:09.370427    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:09.676221 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:09.676174    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:09.993757 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:09.993669    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:10.947587 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:10.947553    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"]
Nov 02 01:57:11.066880 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:11.066849    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"]
Nov 02 01:57:11.348642 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:11.348542    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:11.581751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:11.581688    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:12.139614 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:57:12.139580    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:57:15.137489 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:15.137455    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:15.894981 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:15.894949    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:15.947966 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:15.947930    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8" podUID="166ad525-f537-4c70-8041-af4f04d0f0dc" containerName="payload-processing" containerID="cri-o://152691f186fe62cbb7c5cf46fb048bc0484717f88239751252003aff292b055a" gracePeriod=25
Nov 02 01:57:16.068198 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.068151    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d" podUID="d9c42d23-3286-4128-aeb7-42bc8e487599" containerName="payload-pre-processing" containerID="cri-o://ad0a280c54eb08c62da107ac23e7859b270132cb175dbc8915dd8b07f7de4944" gracePeriod=25
Nov 02 01:57:16.356384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.346221    4912 generic.go:356] "Generic (PLEG): container finished" podID="166ad525-f537-4c70-8041-af4f04d0f0dc" containerID="152691f186fe62cbb7c5cf46fb048bc0484717f88239751252003aff292b055a" exitCode=0
Nov 02 01:57:16.356384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.346638    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8" event={"ID":"166ad525-f537-4c70-8041-af4f04d0f0dc","Type":"ContainerDied","Data":"152691f186fe62cbb7c5cf46fb048bc0484717f88239751252003aff292b055a"}
Nov 02 01:57:16.356384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.347251    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8" event={"ID":"166ad525-f537-4c70-8041-af4f04d0f0dc","Type":"ContainerDied","Data":"fdfdd2492fcc41469b4b7c12ee1456ae3b4b017fc941badc20fa8eb700a6a7c1"}
Nov 02 01:57:16.356384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.347290    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdfdd2492fcc41469b4b7c12ee1456ae3b4b017fc941badc20fa8eb700a6a7c1"
Nov 02 01:57:16.356384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.350460    4912 generic.go:356] "Generic (PLEG): container finished" podID="d9c42d23-3286-4128-aeb7-42bc8e487599" containerID="ad0a280c54eb08c62da107ac23e7859b270132cb175dbc8915dd8b07f7de4944" exitCode=0
Nov 02 01:57:16.356384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.350512    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d" event={"ID":"d9c42d23-3286-4128-aeb7-42bc8e487599","Type":"ContainerDied","Data":"ad0a280c54eb08c62da107ac23e7859b270132cb175dbc8915dd8b07f7de4944"}
Nov 02 01:57:16.356384 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.352656    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:57:16.534133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.529915    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166ad525-f537-4c70-8041-af4f04d0f0dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166ad525-f537-4c70-8041-af4f04d0f0dc-plugins-config-volume\") pod \"166ad525-f537-4c70-8041-af4f04d0f0dc\" (UID: \"166ad525-f537-4c70-8041-af4f04d0f0dc\") "
Nov 02 01:57:16.534133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.530200    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166ad525-f537-4c70-8041-af4f04d0f0dc-kube-api-access-g4btb\" (UniqueName: \"kubernetes.io/projected/166ad525-f537-4c70-8041-af4f04d0f0dc-kube-api-access-g4btb\") pod \"166ad525-f537-4c70-8041-af4f04d0f0dc\" (UID: \"166ad525-f537-4c70-8041-af4f04d0f0dc\") "
Nov 02 01:57:16.534133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.533277    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166ad525-f537-4c70-8041-af4f04d0f0dc-plugins-config-volume" pod "166ad525-f537-4c70-8041-af4f04d0f0dc" (UID: "166ad525-f537-4c70-8041-af4f04d0f0dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:16.534133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.533892    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166ad525-f537-4c70-8041-af4f04d0f0dc-kube-api-access-g4btb" pod "166ad525-f537-4c70-8041-af4f04d0f0dc" (UID: "166ad525-f537-4c70-8041-af4f04d0f0dc"). InnerVolumeSpecName "kube-api-access-g4btb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:16.535923 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.535900    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:57:16.632323 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.632292    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c42d23-3286-4128-aeb7-42bc8e487599-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c42d23-3286-4128-aeb7-42bc8e487599-plugins-config-volume\") pod \"d9c42d23-3286-4128-aeb7-42bc8e487599\" (UID: \"d9c42d23-3286-4128-aeb7-42bc8e487599\") "
Nov 02 01:57:16.632503 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.632477    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c42d23-3286-4128-aeb7-42bc8e487599-kube-api-access-6klgp\" (UniqueName: \"kubernetes.io/projected/d9c42d23-3286-4128-aeb7-42bc8e487599-kube-api-access-6klgp\") pod \"d9c42d23-3286-4128-aeb7-42bc8e487599\" (UID: \"d9c42d23-3286-4128-aeb7-42bc8e487599\") "
Nov 02 01:57:16.632697 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.632671    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c42d23-3286-4128-aeb7-42bc8e487599-plugins-config-volume" pod "d9c42d23-3286-4128-aeb7-42bc8e487599" (UID: "d9c42d23-3286-4128-aeb7-42bc8e487599"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:16.633070 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.633037    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c42d23-3286-4128-aeb7-42bc8e487599-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:16.633140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.633078    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166ad525-f537-4c70-8041-af4f04d0f0dc-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:16.633140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.633092    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4btb\" (UniqueName: \"kubernetes.io/projected/166ad525-f537-4c70-8041-af4f04d0f0dc-kube-api-access-g4btb\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:16.634645 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.634620    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c42d23-3286-4128-aeb7-42bc8e487599-kube-api-access-6klgp" pod "d9c42d23-3286-4128-aeb7-42bc8e487599" (UID: "d9c42d23-3286-4128-aeb7-42bc8e487599"). InnerVolumeSpecName "kube-api-access-6klgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:16.734812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:16.734771    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6klgp\" (UniqueName: \"kubernetes.io/projected/d9c42d23-3286-4128-aeb7-42bc8e487599-kube-api-access-6klgp\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:17.362432 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:17.362397    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"
Nov 02 01:57:17.362432 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:17.362416    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d" event={"ID":"d9c42d23-3286-4128-aeb7-42bc8e487599","Type":"ContainerDied","Data":"d0ced8fcb7c7e4e0be2b60505370c0d80d6b47dce3e848040bbe0228f92a931a"}
Nov 02 01:57:17.362966 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:17.362494    4912 scope.go:122] "RemoveContainer" containerID="ad0a280c54eb08c62da107ac23e7859b270132cb175dbc8915dd8b07f7de4944"
Nov 02 01:57:17.362966 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:17.362735    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"
Nov 02 01:57:17.529769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:17.529681    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"]
Nov 02 01:57:17.564461 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:17.564429    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-495b48c2-65bc69479-k575d"]
Nov 02 01:57:17.641575 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:17.641540    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"]
Nov 02 01:57:17.646607 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:17.646584    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-495b48c2-685cd8f949-h9ds8"]
Nov 02 01:57:19.230544 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:19.230103    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166ad525-f537-4c70-8041-af4f04d0f0dc" path="/var/lib/kubelet/pods/166ad525-f537-4c70-8041-af4f04d0f0dc/volumes"
Nov 02 01:57:19.231861 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:19.231592    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c42d23-3286-4128-aeb7-42bc8e487599" path="/var/lib/kubelet/pods/d9c42d23-3286-4128-aeb7-42bc8e487599/volumes"
Nov 02 01:57:20.040125 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:20.040092    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:21.536883 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:21.536842    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"]
Nov 02 01:57:21.867290 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:21.867256    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:22.040558 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:22.040524    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:24.941432 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:24.941390    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:26.537607 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:26.537560    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7" podUID="089666e8-41da-405d-a8d9-11b81fcb699b" containerName="payload-pre-processing" containerID="cri-o://27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859" gracePeriod=25
Nov 02 01:57:26.842423 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:26.842393    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:57:26.907607 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:26.907573    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089666e8-41da-405d-a8d9-11b81fcb699b-kube-api-access-7vtff\" (UniqueName: \"kubernetes.io/projected/089666e8-41da-405d-a8d9-11b81fcb699b-kube-api-access-7vtff\") pod \"089666e8-41da-405d-a8d9-11b81fcb699b\" (UID: \"089666e8-41da-405d-a8d9-11b81fcb699b\") "
Nov 02 01:57:26.907779 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:26.907749    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089666e8-41da-405d-a8d9-11b81fcb699b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089666e8-41da-405d-a8d9-11b81fcb699b-plugins-config-volume\") pod \"089666e8-41da-405d-a8d9-11b81fcb699b\" (UID: \"089666e8-41da-405d-a8d9-11b81fcb699b\") "
Nov 02 01:57:26.908208 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:26.908182    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089666e8-41da-405d-a8d9-11b81fcb699b-plugins-config-volume" pod "089666e8-41da-405d-a8d9-11b81fcb699b" (UID: "089666e8-41da-405d-a8d9-11b81fcb699b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:26.908433 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:26.908412    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089666e8-41da-405d-a8d9-11b81fcb699b-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:26.909875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:26.909855    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089666e8-41da-405d-a8d9-11b81fcb699b-kube-api-access-7vtff" pod "089666e8-41da-405d-a8d9-11b81fcb699b" (UID: "089666e8-41da-405d-a8d9-11b81fcb699b"). InnerVolumeSpecName "kube-api-access-7vtff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:27.010775 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.010738    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vtff\" (UniqueName: \"kubernetes.io/projected/089666e8-41da-405d-a8d9-11b81fcb699b-kube-api-access-7vtff\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:27.452014 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.451958    4912 generic.go:356] "Generic (PLEG): container finished" podID="089666e8-41da-405d-a8d9-11b81fcb699b" containerID="27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859" exitCode=0
Nov 02 01:57:27.452014 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.452011    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"
Nov 02 01:57:27.452312 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.452051    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7" event={"ID":"089666e8-41da-405d-a8d9-11b81fcb699b","Type":"ContainerDied","Data":"27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859"}
Nov 02 01:57:27.452312 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.452147    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7" event={"ID":"089666e8-41da-405d-a8d9-11b81fcb699b","Type":"ContainerDied","Data":"495c5dd57b705f1618199577ba1803384da0e1a28c92fb74b98e82f963f60451"}
Nov 02 01:57:27.452312 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.452177    4912 scope.go:122] "RemoveContainer" containerID="27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859"
Nov 02 01:57:27.466916 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.466896    4912 scope.go:122] "RemoveContainer" containerID="27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859"
Nov 02 01:57:27.467225 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:57:27.467204    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859\": container with ID starting with 27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859 not found: ID does not exist" containerID="27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859"
Nov 02 01:57:27.467291 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.467232    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859"} err="failed to get container status \"27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859\": rpc error: code = NotFound desc = could not find container \"27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859\": container with ID starting with 27d1ce9e6fc516ab6fb12453389911fc73ecb082643b8ba4148e99f60d54c859 not found: ID does not exist"
Nov 02 01:57:27.619140 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.619107    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"]
Nov 02 01:57:27.668628 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:27.668597    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9a2fcb6-758796d8c-zbrh7"]
Nov 02 01:57:29.228568 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:29.228530    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089666e8-41da-405d-a8d9-11b81fcb699b" path="/var/lib/kubelet/pods/089666e8-41da-405d-a8d9-11b81fcb699b/volumes"
Nov 02 01:57:29.744465 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:29.744417    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:31.176217 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.176185    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"]
Nov 02 01:57:31.177854 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177823    4912 cpu_manager.go:414] "containerMap: removing container" podUID="166ad525-f537-4c70-8041-af4f04d0f0dc" containerName="payload-processing"
Nov 02 01:57:31.177854 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177854    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166ad525-f537-4c70-8041-af4f04d0f0dc" containerName="payload-processing"
Nov 02 01:57:31.178015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177884    4912 cpu_manager.go:414] "containerMap: removing container" podUID="e117c74a-26e9-49a4-9429-45b68fd1a634" containerName="payload-pre-processing"
Nov 02 01:57:31.178015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177892    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e117c74a-26e9-49a4-9429-45b68fd1a634" containerName="payload-pre-processing"
Nov 02 01:57:31.178015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177915    4912 cpu_manager.go:414] "containerMap: removing container" podUID="089666e8-41da-405d-a8d9-11b81fcb699b" containerName="payload-pre-processing"
Nov 02 01:57:31.178015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177924    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089666e8-41da-405d-a8d9-11b81fcb699b" containerName="payload-pre-processing"
Nov 02 01:57:31.178015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177970    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d9c42d23-3286-4128-aeb7-42bc8e487599" containerName="payload-pre-processing"
Nov 02 01:57:31.178015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177978    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c42d23-3286-4128-aeb7-42bc8e487599" containerName="payload-pre-processing"
Nov 02 01:57:31.178015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177991    4912 cpu_manager.go:414] "containerMap: removing container" podUID="2cbbb0cf-795e-4489-a161-f2002e56c3f7" containerName="payload-processing"
Nov 02 01:57:31.178015 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.177999    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbbb0cf-795e-4489-a161-f2002e56c3f7" containerName="payload-processing"
Nov 02 01:57:31.178277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.178181    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c42d23-3286-4128-aeb7-42bc8e487599" containerName="payload-pre-processing"
Nov 02 01:57:31.178277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.178194    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbbb0cf-795e-4489-a161-f2002e56c3f7" containerName="payload-processing"
Nov 02 01:57:31.178277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.178212    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="166ad525-f537-4c70-8041-af4f04d0f0dc" containerName="payload-processing"
Nov 02 01:57:31.178277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.178226    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="e117c74a-26e9-49a4-9429-45b68fd1a634" containerName="payload-pre-processing"
Nov 02 01:57:31.178277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.178238    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="089666e8-41da-405d-a8d9-11b81fcb699b" containerName="payload-pre-processing"
Nov 02 01:57:31.184506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.184482    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.380199 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.380154    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.380402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.380246    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmfz\" (UniqueName: \"kubernetes.io/projected/da8141a6-dfcd-4672-a751-1d2f60d39bed-kube-api-access-4xmfz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.380402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.380306    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.380402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.380337    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.380585 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.380426    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8141a6-dfcd-4672-a751-1d2f60d39bed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.380585 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.380522    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.380585 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.380552    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.380752 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.380634    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.380752 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.380690    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.405997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.405961    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"]
Nov 02 01:57:31.482105 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.481978    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8141a6-dfcd-4672-a751-1d2f60d39bed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.482293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.482117    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.482293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.482151    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.482293 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.482244    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.482657 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.482628    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.482797 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.482711    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.482921 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.482904    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.483380 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.483357    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8141a6-dfcd-4672-a751-1d2f60d39bed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.483496 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.483415    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.483582 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.483545    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmfz\" (UniqueName: \"kubernetes.io/projected/da8141a6-dfcd-4672-a751-1d2f60d39bed-kube-api-access-4xmfz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.483656 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.483635    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.483731 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.483671    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.484036 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.484012    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.484251 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.484217    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.485270 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.485249    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.485413 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.485391    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.543457 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.543424    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8141a6-dfcd-4672-a751-1d2f60d39bed-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.543966 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.543950    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmfz\" (UniqueName: \"kubernetes.io/projected/da8141a6-dfcd-4672-a751-1d2f60d39bed-kube-api-access-4xmfz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-fhsck\" (UID: \"da8141a6-dfcd-4672-a751-1d2f60d39bed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:31.801731 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:31.801633    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:32.251374 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:32.251326    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:57:32.251733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:32.251466    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:57:32.251733 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:32.251558    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:57:32.268392 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:32.268362    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"]
Nov 02 01:57:32.503752 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:32.503674    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" event={"ID":"da8141a6-dfcd-4672-a751-1d2f60d39bed","Type":"ContainerStarted","Data":"a666b6bb2fbbdb10c0ec86e3a02055ab363171e7e62697e39c01185e19ffd136"}
Nov 02 01:57:32.503752 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:32.503727    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" event={"ID":"da8141a6-dfcd-4672-a751-1d2f60d39bed","Type":"ContainerStarted","Data":"1d950ceebd3d5998d38e3f811a2c69c2fc094bbd16cd97670b7eb2ab7bc9a97b"}
Nov 02 01:57:32.624257 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:32.624207    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podStartSLOduration=1.624188945 podStartE2EDuration="1.624188945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:32.622976688 +0000 UTC m=+1545.528442044" watchObservedRunningTime="2026-11-02 01:57:32.624188945 +0000 UTC m=+1545.529654300"
Nov 02 01:57:32.802216 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:32.802136    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:33.803292 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:33.803255    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-fhsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:57:33.803823 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:33.803333    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podUID="da8141a6-dfcd-4672-a751-1d2f60d39bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 01:57:33.865634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:33.858121    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"]
Nov 02 01:57:33.865634 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:33.858422    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq" podUID="903ce585-e857-4264-a82c-af2affadc6e3" containerName="maas-api" containerID="cri-o://afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66" gracePeriod=30
Nov 02 01:57:33.991811 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:33.991773    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:34.327812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.325304    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:57:34.337672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.337646    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-metrics-certs\") pod \"903ce585-e857-4264-a82c-af2affadc6e3\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") "
Nov 02 01:57:34.337831 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.337696    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903ce585-e857-4264-a82c-af2affadc6e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/903ce585-e857-4264-a82c-af2affadc6e3-openshift-service-ca\") pod \"903ce585-e857-4264-a82c-af2affadc6e3\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") "
Nov 02 01:57:34.337920 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.337900    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903ce585-e857-4264-a82c-af2affadc6e3-kube-api-access-vllx2\" (UniqueName: \"kubernetes.io/projected/903ce585-e857-4264-a82c-af2affadc6e3-kube-api-access-vllx2\") pod \"903ce585-e857-4264-a82c-af2affadc6e3\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") "
Nov 02 01:57:34.337987 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.337958    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls\") pod \"903ce585-e857-4264-a82c-af2affadc6e3\" (UID: \"903ce585-e857-4264-a82c-af2affadc6e3\") "
Nov 02 01:57:34.338102 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.338079    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903ce585-e857-4264-a82c-af2affadc6e3-openshift-service-ca" pod "903ce585-e857-4264-a82c-af2affadc6e3" (UID: "903ce585-e857-4264-a82c-af2affadc6e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:34.338534 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.338510    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/903ce585-e857-4264-a82c-af2affadc6e3-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:34.340001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.339974    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-metrics-certs" pod "903ce585-e857-4264-a82c-af2affadc6e3" (UID: "903ce585-e857-4264-a82c-af2affadc6e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:57:34.340165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.340098    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls" pod "903ce585-e857-4264-a82c-af2affadc6e3" (UID: "903ce585-e857-4264-a82c-af2affadc6e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:57:34.340215 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.340180    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903ce585-e857-4264-a82c-af2affadc6e3-kube-api-access-vllx2" pod "903ce585-e857-4264-a82c-af2affadc6e3" (UID: "903ce585-e857-4264-a82c-af2affadc6e3"). InnerVolumeSpecName "kube-api-access-vllx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:34.440155 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.439956    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:34.440155 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.439990    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/903ce585-e857-4264-a82c-af2affadc6e3-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:34.440155 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.440000    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vllx2\" (UniqueName: \"kubernetes.io/projected/903ce585-e857-4264-a82c-af2affadc6e3-kube-api-access-vllx2\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:34.530212 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.525640    4912 generic.go:356] "Generic (PLEG): container finished" podID="903ce585-e857-4264-a82c-af2affadc6e3" containerID="afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66" exitCode=0
Nov 02 01:57:34.530212 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.526549    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"
Nov 02 01:57:34.530212 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.526876    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq" event={"ID":"903ce585-e857-4264-a82c-af2affadc6e3","Type":"ContainerDied","Data":"afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66"}
Nov 02 01:57:34.530212 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.526941    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq" event={"ID":"903ce585-e857-4264-a82c-af2affadc6e3","Type":"ContainerDied","Data":"b1a32929f7b8f9ec16ffa2aab8395f68d23e4c4631d13aa48ab11c00aba8ff2c"}
Nov 02 01:57:34.530212 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.526969    4912 scope.go:122] "RemoveContainer" containerID="afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66"
Nov 02 01:57:34.584768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.584248    4912 scope.go:122] "RemoveContainer" containerID="afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66"
Nov 02 01:57:34.584768 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:57:34.584659    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66\": container with ID starting with afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66 not found: ID does not exist" containerID="afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66"
Nov 02 01:57:34.584768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.584697    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66"} err="failed to get container status \"afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66\": rpc error: code = NotFound desc = could not find container \"afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66\": container with ID starting with afb0332b6d5830c6e1b12818e79a8d26bc6c38af6ac885549a3aa2190c7d4c66 not found: ID does not exist"
Nov 02 01:57:34.696165 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.696051    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"]
Nov 02 01:57:34.703376 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.703346    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15487453-75d7b55876-5dsnq"]
Nov 02 01:57:34.802901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.802866    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-fhsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:57:34.803084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:34.802927    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podUID="da8141a6-dfcd-4672-a751-1d2f60d39bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 01:57:35.228840 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:35.228813    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903ce585-e857-4264-a82c-af2affadc6e3" path="/var/lib/kubelet/pods/903ce585-e857-4264-a82c-af2affadc6e3/volumes"
Nov 02 01:57:35.802230 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:35.802196    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-fhsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:57:35.802416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:35.802249    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podUID="da8141a6-dfcd-4672-a751-1d2f60d39bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 01:57:35.823446 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:35.823415    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:36.117816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:36.117784    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:36.327557 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:36.327523    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:36.602082 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:36.601975    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:36.822080 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:36.805695    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-fhsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:57:36.822080 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:36.805755    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podUID="da8141a6-dfcd-4672-a751-1d2f60d39bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 01:57:37.203688 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:37.203656    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:37.805763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:37.802477    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-fhsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:57:37.805763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:37.802538    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podUID="da8141a6-dfcd-4672-a751-1d2f60d39bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 01:57:37.921081 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:37.920098    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:38.700444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:38.700398    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:38.802743 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:38.802691    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-fhsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:57:38.802923 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:38.802776    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podUID="da8141a6-dfcd-4672-a751-1d2f60d39bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 01:57:39.802329 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:39.802290    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-fhsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 01:57:39.802773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:39.802358    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podUID="da8141a6-dfcd-4672-a751-1d2f60d39bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 01:57:40.053625 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:40.053537    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"]
Nov 02 01:57:40.104678 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:40.104646    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3025d6f1-2pkxh"]
Nov 02 01:57:40.810758 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:40.810729    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:40.811197 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:40.811021    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:40.811862 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:40.811842    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck"
Nov 02 01:57:41.230476 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.230441    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96772f5-05c8-49f7-9c31-1fccda98f0f7" path="/var/lib/kubelet/pods/a96772f5-05c8-49f7-9c31-1fccda98f0f7/volumes"
Nov 02 01:57:41.311798 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.311768    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"]
Nov 02 01:57:41.311998 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.311954    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k" podUID="f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" containerName="maas-api" containerID="cri-o://8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1" gracePeriod=30
Nov 02 01:57:41.543623 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.543587    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:41.654632 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.654602    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:57:41.655680 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.655648    4912 generic.go:356] "Generic (PLEG): container finished" podID="f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" containerID="8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1" exitCode=0
Nov 02 01:57:41.655885 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.655862    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k" event={"ID":"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae","Type":"ContainerDied","Data":"8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1"}
Nov 02 01:57:41.655963 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.655918    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k" event={"ID":"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae","Type":"ContainerDied","Data":"da8c32c05a8a80fa190f73044099f39fe0a3b8aafd89a025fafa247254f781e8"}
Nov 02 01:57:41.655963 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.655949    4912 scope.go:122] "RemoveContainer" containerID="8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1"
Nov 02 01:57:41.674821 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.674803    4912 scope.go:122] "RemoveContainer" containerID="8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1"
Nov 02 01:57:41.675160 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:57:41.675135    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1\": container with ID starting with 8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1 not found: ID does not exist" containerID="8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1"
Nov 02 01:57:41.675273 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.675167    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1"} err="failed to get container status \"8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1\": rpc error: code = NotFound desc = could not find container \"8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1\": container with ID starting with 8db107cc3a45253386fd489c5e3715aa6e4a40be6023fd9d7739ceec38c5b5a1 not found: ID does not exist"
Nov 02 01:57:41.760424 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.760333    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:41.785836 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.785803    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-kube-api-access-8c2mp\" (UniqueName: \"kubernetes.io/projected/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-kube-api-access-8c2mp\") pod \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") "
Nov 02 01:57:41.786032 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.786004    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-metrics-certs\") pod \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") "
Nov 02 01:57:41.786431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.786131    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-openshift-service-ca\") pod \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") "
Nov 02 01:57:41.786431 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.786157    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls\") pod \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\" (UID: \"f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae\") "
Nov 02 01:57:41.786614 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.786487    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-openshift-service-ca" pod "f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" (UID: "f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:41.787519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.786934    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:41.789753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.789728    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-metrics-certs" pod "f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" (UID: "f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:57:41.789753 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.789737    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls" pod "f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" (UID: "f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:57:41.794437 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.794410    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-kube-api-access-8c2mp" pod "f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" (UID: "f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae"). InnerVolumeSpecName "kube-api-access-8c2mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:41.888729 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.888679    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:41.888729 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.888723    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:41.889308 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.888738    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c2mp\" (UniqueName: \"kubernetes.io/projected/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae-kube-api-access-8c2mp\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:41.943284 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:41.943167    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:42.151874 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:57:42.151836    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:57:42.681386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:42.681357    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"
Nov 02 01:57:42.848276 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:42.848244    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"]
Nov 02 01:57:42.898145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:42.898108    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-495b48c2-59cbc46d56-5g28k"]
Nov 02 01:57:43.227624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:43.227588    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" path="/var/lib/kubelet/pods/f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae/volumes"
Nov 02 01:57:43.862507 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:43.862464    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:44.577774 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:44.577736    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:44.862689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:44.862593    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:45.035463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.035426    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:45.213918 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.213884    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:45.526610 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.526523    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"]
Nov 02 01:57:45.527402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.527382    4912 cpu_manager.go:414] "containerMap: removing container" podUID="903ce585-e857-4264-a82c-af2affadc6e3" containerName="maas-api"
Nov 02 01:57:45.527402 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.527402    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903ce585-e857-4264-a82c-af2affadc6e3" containerName="maas-api"
Nov 02 01:57:45.527595 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.527434    4912 cpu_manager.go:414] "containerMap: removing container" podUID="f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" containerName="maas-api"
Nov 02 01:57:45.527595 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.527440    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" containerName="maas-api"
Nov 02 01:57:45.527703 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.527666    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="903ce585-e857-4264-a82c-af2affadc6e3" containerName="maas-api"
Nov 02 01:57:45.527703 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.527686    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="f709d20a-c6bb-44eb-aa3a-4f0d1c0b80ae" containerName="maas-api"
Nov 02 01:57:45.532860 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.532838    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:57:45.592716 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.592681    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"]
Nov 02 01:57:45.634307 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.634276    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4f2f926d\""
Nov 02 01:57:45.635922 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.634672    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4f2f926d-dockercfg-jqdr9\""
Nov 02 01:57:45.691624 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.691312    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-plugins-config-volume\") pod \"payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66\" (UID: \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\") " pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:57:45.692458 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.692431    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddmq\" (UniqueName: \"kubernetes.io/projected/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-kube-api-access-fddmq\") pod \"payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66\" (UID: \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\") " pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:57:45.711577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.711538    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"]
Nov 02 01:57:45.720782 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.720743    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:45.727319 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.727294    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:45.807360 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.807265    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddmq\" (UniqueName: \"kubernetes.io/projected/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-kube-api-access-fddmq\") pod \"payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66\" (UID: \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\") " pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:57:45.809392 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.809348    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03444ac3-12af-4fa0-b676-56ed9b7d67cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8\" (UID: \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:45.810319 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.810284    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mhbk\" (UniqueName: \"kubernetes.io/projected/03444ac3-12af-4fa0-b676-56ed9b7d67cb-kube-api-access-5mhbk\") pod \"payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8\" (UID: \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:45.810976 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.810955    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-plugins-config-volume\") pod \"payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66\" (UID: \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\") " pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:57:45.816577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.816094    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-plugins-config-volume\") pod \"payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66\" (UID: \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\") " pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:57:45.851720 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.851689    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"]
Nov 02 01:57:45.908013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.907974    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddmq\" (UniqueName: \"kubernetes.io/projected/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-kube-api-access-fddmq\") pod \"payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66\" (UID: \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\") " pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:57:45.915881 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.914900    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03444ac3-12af-4fa0-b676-56ed9b7d67cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8\" (UID: \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:45.916194 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.915845    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03444ac3-12af-4fa0-b676-56ed9b7d67cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8\" (UID: \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:45.917479 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.916494    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mhbk\" (UniqueName: \"kubernetes.io/projected/03444ac3-12af-4fa0-b676-56ed9b7d67cb-kube-api-access-5mhbk\") pod \"payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8\" (UID: \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:45.978532 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:45.978503    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mhbk\" (UniqueName: \"kubernetes.io/projected/03444ac3-12af-4fa0-b676-56ed9b7d67cb-kube-api-access-5mhbk\") pod \"payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8\" (UID: \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:46.044497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.044452    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:46.153961 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.151011    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:57:46.398204 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.397942    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:46.504355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.504294    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"]
Nov 02 01:57:46.504997 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:57:46.504967    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03444ac3_12af_4fa0_b676_56ed9b7d67cb.slice/crio-a6e0df8406b292d62a856c3ced3ae587868c3c9e9d1e96fdc9706265b6d0bb06 WatchSource:0}: Error finding container a6e0df8406b292d62a856c3ced3ae587868c3c9e9d1e96fdc9706265b6d0bb06: Status 404 returned error can't find the container with id a6e0df8406b292d62a856c3ced3ae587868c3c9e9d1e96fdc9706265b6d0bb06
Nov 02 01:57:46.570209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.570181    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:46.723103 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:57:46.723032    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83f3127_e413_4b06_ab1d_a14cf7d65ff4.slice/crio-51c5aeb31be84b872afe10d67b251ca0178dacec0d06ec70a8c0599204c38dc6 WatchSource:0}: Error finding container 51c5aeb31be84b872afe10d67b251ca0178dacec0d06ec70a8c0599204c38dc6: Status 404 returned error can't find the container with id 51c5aeb31be84b872afe10d67b251ca0178dacec0d06ec70a8c0599204c38dc6
Nov 02 01:57:46.724362 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.724311    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"]
Nov 02 01:57:46.748762 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.748719    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66" event={"ID":"b83f3127-e413-4b06-ab1d-a14cf7d65ff4","Type":"ContainerStarted","Data":"51c5aeb31be84b872afe10d67b251ca0178dacec0d06ec70a8c0599204c38dc6"}
Nov 02 01:57:46.751869 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.751833    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8" event={"ID":"03444ac3-12af-4fa0-b676-56ed9b7d67cb","Type":"ContainerStarted","Data":"a1ecf5489f2285b12a330c519d440a832dd9e94844e3cd0ca681077ffd9221ad"}
Nov 02 01:57:46.752467 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.751914    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8" event={"ID":"03444ac3-12af-4fa0-b676-56ed9b7d67cb","Type":"ContainerStarted","Data":"a6e0df8406b292d62a856c3ced3ae587868c3c9e9d1e96fdc9706265b6d0bb06"}
Nov 02 01:57:46.752467 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.751974    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:46.872155 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:46.872103    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8" podStartSLOduration=1.872086533 podStartE2EDuration="1.872086533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:46.866763809 +0000 UTC m=+1559.772229164" watchObservedRunningTime="2026-11-02 01:57:46.872086533 +0000 UTC m=+1559.777551895"
Nov 02 01:57:47.392728 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:47.392692    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:47.709954 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:47.709854    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:47.766001 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:47.765962    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66" event={"ID":"b83f3127-e413-4b06-ab1d-a14cf7d65ff4","Type":"ContainerStarted","Data":"a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160"}
Nov 02 01:57:47.766490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:47.766034    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:57:47.884181 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:47.884132    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66" podStartSLOduration=2.884117702 podStartE2EDuration="2.884117702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:47.880333891 +0000 UTC m=+1560.785799250" watchObservedRunningTime="2026-11-02 01:57:47.884117702 +0000 UTC m=+1560.789583058"
Nov 02 01:57:48.674834 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:48.674792    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"]
Nov 02 01:57:48.675154 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:48.675093    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn" podUID="c95b99c5-0fbc-4c50-8611-20e3cbb665df" containerName="maas-api" containerID="cri-o://cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81" gracePeriod=30
Nov 02 01:57:48.803893 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:48.803847    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:49.127571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.127543    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:57:49.295788 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.295751    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-metrics-certs\") pod \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") "
Nov 02 01:57:49.295953 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.295815    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95b99c5-0fbc-4c50-8611-20e3cbb665df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95b99c5-0fbc-4c50-8611-20e3cbb665df-openshift-service-ca\") pod \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") "
Nov 02 01:57:49.295953 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.295869    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95b99c5-0fbc-4c50-8611-20e3cbb665df-kube-api-access-9pbw8\" (UniqueName: \"kubernetes.io/projected/c95b99c5-0fbc-4c50-8611-20e3cbb665df-kube-api-access-9pbw8\") pod \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") "
Nov 02 01:57:49.296638 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.296612    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95b99c5-0fbc-4c50-8611-20e3cbb665df-openshift-service-ca" pod "c95b99c5-0fbc-4c50-8611-20e3cbb665df" (UID: "c95b99c5-0fbc-4c50-8611-20e3cbb665df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:49.298335 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.298311    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls\") pod \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\" (UID: \"c95b99c5-0fbc-4c50-8611-20e3cbb665df\") "
Nov 02 01:57:49.299575 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.299552    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95b99c5-0fbc-4c50-8611-20e3cbb665df-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:49.362049 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.361960    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls" pod "c95b99c5-0fbc-4c50-8611-20e3cbb665df" (UID: "c95b99c5-0fbc-4c50-8611-20e3cbb665df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:57:49.364215 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.364179    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-metrics-certs" pod "c95b99c5-0fbc-4c50-8611-20e3cbb665df" (UID: "c95b99c5-0fbc-4c50-8611-20e3cbb665df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:57:49.364765 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.364734    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95b99c5-0fbc-4c50-8611-20e3cbb665df-kube-api-access-9pbw8" pod "c95b99c5-0fbc-4c50-8611-20e3cbb665df" (UID: "c95b99c5-0fbc-4c50-8611-20e3cbb665df"). InnerVolumeSpecName "kube-api-access-9pbw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:49.400781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.400748    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:49.400781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.400777    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbw8\" (UniqueName: \"kubernetes.io/projected/c95b99c5-0fbc-4c50-8611-20e3cbb665df-kube-api-access-9pbw8\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:49.400781 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.400789    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c95b99c5-0fbc-4c50-8611-20e3cbb665df-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:49.791938 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.791892    4912 generic.go:356] "Generic (PLEG): container finished" podID="c95b99c5-0fbc-4c50-8611-20e3cbb665df" containerID="cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81" exitCode=0
Nov 02 01:57:49.791938 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.791937    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"
Nov 02 01:57:49.792241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.791986    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn" event={"ID":"c95b99c5-0fbc-4c50-8611-20e3cbb665df","Type":"ContainerDied","Data":"cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81"}
Nov 02 01:57:49.792241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.792048    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn" event={"ID":"c95b99c5-0fbc-4c50-8611-20e3cbb665df","Type":"ContainerDied","Data":"1994459f802bde1a655dcbaada9ddac523951e3d8d6537b809054d26ca494158"}
Nov 02 01:57:49.792241 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.792098    4912 scope.go:122] "RemoveContainer" containerID="cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81"
Nov 02 01:57:49.806594 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.806457    4912 scope.go:122] "RemoveContainer" containerID="cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81"
Nov 02 01:57:49.806852 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:57:49.806742    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81\": container with ID starting with cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81 not found: ID does not exist" containerID="cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81"
Nov 02 01:57:49.806852 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.806770    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81"} err="failed to get container status \"cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81\": rpc error: code = NotFound desc = could not find container \"cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81\": container with ID starting with cffd323f867503ac0bfdbdcc9abf932d02217065c2dd1c69d1b64446d6d07b81 not found: ID does not exist"
Nov 02 01:57:49.962796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:49.962758    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"]
Nov 02 01:57:50.010075 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:50.010021    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:50.012430 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:50.012403    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9a2fcb6-565554c6ff-knnpn"]
Nov 02 01:57:50.194188 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:50.194153    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:50.497816 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:50.497720    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:50.748005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:50.747913    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:50.932945 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:50.932899    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:51.229213 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:51.229182    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95b99c5-0fbc-4c50-8611-20e3cbb665df" path="/var/lib/kubelet/pods/c95b99c5-0fbc-4c50-8611-20e3cbb665df/volumes"
Nov 02 01:57:51.341118 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:51.333120    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:51.521706 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:51.521614    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:51.767212 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:51.767181    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:51.943802 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:51.943761    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:52.191246 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:52.191184    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:52.380894 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:52.380853    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:54.815209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:54.815179    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:54.998513 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:54.998478    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:55.351013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.350982    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"]
Nov 02 01:57:55.351790 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.351775    4912 cpu_manager.go:414] "containerMap: removing container" podUID="c95b99c5-0fbc-4c50-8611-20e3cbb665df" containerName="maas-api"
Nov 02 01:57:55.351900 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.351792    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95b99c5-0fbc-4c50-8611-20e3cbb665df" containerName="maas-api"
Nov 02 01:57:55.351940 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.351922    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95b99c5-0fbc-4c50-8611-20e3cbb665df" containerName="maas-api"
Nov 02 01:57:55.357254 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.357230    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:57:55.413096 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.413045    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"]
Nov 02 01:57:55.430355 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.430314    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1588de9-edb4-4d91-8b03-40d02e25c524-plugins-config-volume\") pod \"payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx\" (UID: \"b1588de9-edb4-4d91-8b03-40d02e25c524\") " pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:57:55.430646 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.430622    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgb2\" (UniqueName: \"kubernetes.io/projected/b1588de9-edb4-4d91-8b03-40d02e25c524-kube-api-access-gdgb2\") pod \"payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx\" (UID: \"b1588de9-edb4-4d91-8b03-40d02e25c524\") " pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:57:55.455639 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.455600    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-af807eae-dockercfg-tst9n\""
Nov 02 01:57:55.455845 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.455721    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-af807eae\""
Nov 02 01:57:55.530203 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.530172    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"]
Nov 02 01:57:55.532977 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.532946    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1588de9-edb4-4d91-8b03-40d02e25c524-plugins-config-volume\") pod \"payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx\" (UID: \"b1588de9-edb4-4d91-8b03-40d02e25c524\") " pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:57:55.533178 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.533158    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgb2\" (UniqueName: \"kubernetes.io/projected/b1588de9-edb4-4d91-8b03-40d02e25c524-kube-api-access-gdgb2\") pod \"payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx\" (UID: \"b1588de9-edb4-4d91-8b03-40d02e25c524\") " pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:57:55.533726 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.533703    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1588de9-edb4-4d91-8b03-40d02e25c524-plugins-config-volume\") pod \"payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx\" (UID: \"b1588de9-edb4-4d91-8b03-40d02e25c524\") " pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:57:55.535737 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.535718    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:57:55.602478 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.602365    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"]
Nov 02 01:57:55.605878 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.605241    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgb2\" (UniqueName: \"kubernetes.io/projected/b1588de9-edb4-4d91-8b03-40d02e25c524-kube-api-access-gdgb2\") pod \"payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx\" (UID: \"b1588de9-edb4-4d91-8b03-40d02e25c524\") " pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:57:55.635052 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.635013    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg\" (UID: \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:57:55.635370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.635344    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrhxt\" (UniqueName: \"kubernetes.io/projected/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-kube-api-access-lrhxt\") pod \"payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg\" (UID: \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:57:55.678905 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.678871    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:57:55.748080 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.747142    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg\" (UID: \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:57:55.748702 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.748675    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg\" (UID: \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:57:55.748887 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.748862    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrhxt\" (UniqueName: \"kubernetes.io/projected/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-kube-api-access-lrhxt\") pod \"payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg\" (UID: \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:57:55.900777 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:55.900741    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrhxt\" (UniqueName: \"kubernetes.io/projected/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-kube-api-access-lrhxt\") pod \"payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg\" (UID: \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:57:56.115492 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:57:56.115455    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1588de9_edb4_4d91_8b03_40d02e25c524.slice/crio-08e126caeb7103ac808595f9545b4fa486de9f25b1767e47396c4eb79c1be65e WatchSource:0}: Error finding container 08e126caeb7103ac808595f9545b4fa486de9f25b1767e47396c4eb79c1be65e: Status 404 returned error can't find the container with id 08e126caeb7103ac808595f9545b4fa486de9f25b1767e47396c4eb79c1be65e
Nov 02 01:57:56.115626 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.115521    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"]
Nov 02 01:57:56.152970 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.152942    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:57:56.529552 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.529455    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:57:56.564344 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.564313    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"]
Nov 02 01:57:56.569707 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:57:56.569674    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7ca09a_55ac_46fa_9e98_c5b33e3228d4.slice/crio-20c8fff033ab0c61accddd25865c3c8c942e4ee776e62308a97b43b78b698289 WatchSource:0}: Error finding container 20c8fff033ab0c61accddd25865c3c8c942e4ee776e62308a97b43b78b698289: Status 404 returned error can't find the container with id 20c8fff033ab0c61accddd25865c3c8c942e4ee776e62308a97b43b78b698289
Nov 02 01:57:56.938654 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.938613    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg" event={"ID":"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4","Type":"ContainerStarted","Data":"8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f"}
Nov 02 01:57:56.939255 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.938677    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg" event={"ID":"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4","Type":"ContainerStarted","Data":"20c8fff033ab0c61accddd25865c3c8c942e4ee776e62308a97b43b78b698289"}
Nov 02 01:57:56.939255 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.938710    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:57:56.940607 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.940582    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx" event={"ID":"b1588de9-edb4-4d91-8b03-40d02e25c524","Type":"ContainerStarted","Data":"b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78"}
Nov 02 01:57:56.940743 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.940625    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx" event={"ID":"b1588de9-edb4-4d91-8b03-40d02e25c524","Type":"ContainerStarted","Data":"08e126caeb7103ac808595f9545b4fa486de9f25b1767e47396c4eb79c1be65e"}
Nov 02 01:57:56.940743 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:56.940651    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:57:57.057376 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:57.057325    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg" podStartSLOduration=2.057308517 podStartE2EDuration="2.057308517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:57.053426983 +0000 UTC m=+1569.958892342" watchObservedRunningTime="2026-11-02 01:57:57.057308517 +0000 UTC m=+1569.962773871"
Nov 02 01:57:57.181683 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:57.181586    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx" podStartSLOduration=2.181566748 podStartE2EDuration="2.181566748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:57.171615681 +0000 UTC m=+1570.077081041" watchObservedRunningTime="2026-11-02 01:57:57.181566748 +0000 UTC m=+1570.087032107"
Nov 02 01:57:57.769091 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:57.769041    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:57:58.784574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:57:58.784542    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:58:01.355740 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:01.355702    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:04.758320 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.758278    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"]
Nov 02 01:58:04.764348 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.764320    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.805468 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.805434    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.805653 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.805476    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.805653 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.805634    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.805778 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.805739    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1d8e1-10e5-4225-bcbe-66653abe617f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.805873 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.805852    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.805936 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.805890    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rvt\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-kube-api-access-c5rvt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.806115 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.806094    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.822220 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.822186    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 01:58:04.822220 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.822206    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 01:58:04.822395 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.822238    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 01:58:04.822395 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.822331    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-f8tcq\""
Nov 02 01:58:04.881183 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.881146    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"]
Nov 02 01:58:04.907166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.907131    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.907377 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.907227    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.907377 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.907258    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.907377 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.907349    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.907574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.907426    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1d8e1-10e5-4225-bcbe-66653abe617f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.907574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.907514    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.907574 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.907562    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rvt\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-kube-api-access-c5rvt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.907964 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.907938    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.910711 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.910692    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.910958 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.910930    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.911301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.911280    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1d8e1-10e5-4225-bcbe-66653abe617f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.913217 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.913193    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.969481 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.969450    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:04.969714 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:04.969694    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rvt\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-kube-api-access-c5rvt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2gcp7\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:05.050307 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:05.050219    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"]
Nov 02 01:58:05.058762 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:05.058726    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50abcf4-7h78b"]
Nov 02 01:58:05.079501 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:05.079467    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:05.230383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:05.230342    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a8aac8-11fe-4c9e-9c01-e472fae4f09c" path="/var/lib/kubelet/pods/01a8aac8-11fe-4c9e-9c01-e472fae4f09c/volumes"
Nov 02 01:58:05.527401 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:58:05.527371    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe1d8e1_10e5_4225_bcbe_66653abe617f.slice/crio-bc74b05f6c87a3015efa8f3fa68bf9a9aedbca818c17cfcfddec076a22084a7c WatchSource:0}: Error finding container bc74b05f6c87a3015efa8f3fa68bf9a9aedbca818c17cfcfddec076a22084a7c: Status 404 returned error can't find the container with id bc74b05f6c87a3015efa8f3fa68bf9a9aedbca818c17cfcfddec076a22084a7c
Nov 02 01:58:05.529095 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:05.529046    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"]
Nov 02 01:58:06.028358 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:06.028323    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7" event={"ID":"1fe1d8e1-10e5-4225-bcbe-66653abe617f","Type":"ContainerStarted","Data":"bc74b05f6c87a3015efa8f3fa68bf9a9aedbca818c17cfcfddec076a22084a7c"}
Nov 02 01:58:06.097761 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:06.097724    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:07.958372 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:07.958339    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 01:58:07.959409 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:07.959390    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 01:58:09.845050 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:09.845016    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:58:09.845393 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:09.845101    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 01:58:10.075688 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:10.075648    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7" event={"ID":"1fe1d8e1-10e5-4225-bcbe-66653abe617f","Type":"ContainerStarted","Data":"e79f137379ef127c1832955bb0a6937f3f3a17bef9802b93ae7fcc7d7ef5d961"}
Nov 02 01:58:10.075909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:10.075721    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:10.078136 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:10.078094    4912 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2gcp7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:58:10.078291 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:10.078150    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7" podUID="1fe1d8e1-10e5-4225-bcbe-66653abe617f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:58:10.251627 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:10.251540    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7" podStartSLOduration=1.937323063 podStartE2EDuration="6.251504419s" totalImagesPullingTime="4.314181356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:10.201828443 +0000 UTC m=+1583.107293799" watchObservedRunningTime="2026-11-02 01:58:10.251504419 +0000 UTC m=+1583.156969773"
Nov 02 01:58:10.942680 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:10.942645    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:11.087187 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:11.087150    4912 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2gcp7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:58:11.087371 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:11.087207    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7" podUID="1fe1d8e1-10e5-4225-bcbe-66653abe617f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:58:12.157453 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:58:12.157414    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:58:13.089663 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:13.089621    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"]
Nov 02 01:58:13.142747 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:13.142693    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbf6de3-9wgxb"]
Nov 02 01:58:13.307085 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:13.299445    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65f2d2f-d1e8-4497-b4be-885b48ddaf2c" path="/var/lib/kubelet/pods/b65f2d2f-d1e8-4497-b4be-885b48ddaf2c/volumes"
Nov 02 01:58:14.115271 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:14.113251    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 01:58:16.244657 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:16.244617    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:18.127785 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:18.127748    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:21.046359 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:21.046318    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:24.164472 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:24.164433    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:25.361039 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:25.360991    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:25.694539 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:25.694497    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:25.925031 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:25.924996    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:26.184504 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:26.184468    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:26.365921 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:26.365876    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:26.684889 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:26.684842    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:31.140379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:31.140342    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:31.438248 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:31.438161    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:31.632847 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:31.632809    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:31.827745 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:31.827641    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:33.827730 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:33.827690    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:34.468712 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:34.468565    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:34.650435 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:34.650300    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:34.826333 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:34.826227    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:35.013166 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.013131    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"]
Nov 02 01:58:35.014530 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.014494    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:35.019960 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.019937    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:35.079993 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.079916    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"]
Nov 02 01:58:35.124306 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.124251    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-73e1cc-dockercfg-z57qx\""
Nov 02 01:58:35.124812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.124769    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-73e1cc\""
Nov 02 01:58:35.132655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.132624    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns\" (UID: \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:35.132999 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.132976    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fd8w\" (UniqueName: \"kubernetes.io/projected/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-kube-api-access-8fd8w\") pod \"payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns\" (UID: \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:35.236715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.236662    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns\" (UID: \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:35.236998 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.236976    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fd8w\" (UniqueName: \"kubernetes.io/projected/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-kube-api-access-8fd8w\") pod \"payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns\" (UID: \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:35.238497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.238452    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns\" (UID: \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:35.299328 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.299291    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fd8w\" (UniqueName: \"kubernetes.io/projected/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-kube-api-access-8fd8w\") pod \"payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns\" (UID: \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:35.369145 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.364361    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:35.803851 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:58:35.803816    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05adf78_f5e8_45dd_87a0_fd8c04cb1c79.slice/crio-a3bd1730f3c718f507335dafb56284a104e4da68a32a7ae55b9430db34590465 WatchSource:0}: Error finding container a3bd1730f3c718f507335dafb56284a104e4da68a32a7ae55b9430db34590465: Status 404 returned error can't find the container with id a3bd1730f3c718f507335dafb56284a104e4da68a32a7ae55b9430db34590465
Nov 02 01:58:35.805448 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:35.805422    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"]
Nov 02 01:58:36.056164 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.056076    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"]
Nov 02 01:58:36.061312 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.061283    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30561773-d579c"]
Nov 02 01:58:36.232345 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.232314    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"]
Nov 02 01:58:36.236950 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.236926    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.273751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.273718    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-openshift-service-ca\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.273929 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.273903    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.274006 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.273990    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-metrics-certs\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.274180 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.274157    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznkk\" (UniqueName: \"kubernetes.io/projected/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-kube-api-access-qznkk\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.298200 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.298172    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-af807eae-serving-cert\""
Nov 02 01:58:36.298341 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.298254    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"]
Nov 02 01:58:36.375796 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.375767    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-openshift-service-ca\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.376008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.375852    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.376008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.375918    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-metrics-certs\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.376008 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.375988    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznkk\" (UniqueName: \"kubernetes.io/projected/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-kube-api-access-qznkk\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.376200 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:58:36.376024    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-serving-cert: secret "maas-api-e2e-derive-af807eae-serving-cert" not found
Nov 02 01:58:36.376200 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:58:36.376111    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls podName:c0d33cf6-0a86-4b65-8c26-bc431f262c3b nodeName:}" failed. No retries permitted until 2026-11-02 01:58:36.876090959 +0000 UTC m=+1609.781556307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls") pod "maas-api-e2e-derive-af807eae-546b45557c-jxf54" (UID: "c0d33cf6-0a86-4b65-8c26-bc431f262c3b") : secret "maas-api-e2e-derive-af807eae-serving-cert" not found
Nov 02 01:58:36.376583 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.376563    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-openshift-service-ca\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.379277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.379252    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-metrics-certs\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.437689 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.437655    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznkk\" (UniqueName: \"kubernetes.io/projected/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-kube-api-access-qznkk\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.457194 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.457159    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:36.466133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.466102    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns" event={"ID":"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79","Type":"ContainerStarted","Data":"201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0"}
Nov 02 01:58:36.466301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.466164    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns" event={"ID":"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79","Type":"ContainerStarted","Data":"a3bd1730f3c718f507335dafb56284a104e4da68a32a7ae55b9430db34590465"}
Nov 02 01:58:36.466301 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.466214    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:36.591693 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.591565    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns" podStartSLOduration=2.591543722 podStartE2EDuration="2.591543722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:36.582911321 +0000 UTC m=+1609.488376676" watchObservedRunningTime="2026-11-02 01:58:36.591543722 +0000 UTC m=+1609.497009080"
Nov 02 01:58:36.638992 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.638895    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:36.809950 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.809920    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:36.886997 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.886962    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:36.898024 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:36.897951    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls\") pod \"maas-api-e2e-derive-af807eae-546b45557c-jxf54\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:37.153101 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:37.152725    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:37.236751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:37.236715    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f17372-513e-4ca1-8ea6-1ed086c350b9" path="/var/lib/kubelet/pods/99f17372-513e-4ca1-8ea6-1ed086c350b9/volumes"
Nov 02 01:58:37.652990 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:58:37.652951    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d33cf6_0a86_4b65_8c26_bc431f262c3b.slice/crio-82f0d37bc9340cce3659cbb568c26976c14a07b273751c7a00a43ba2f2eae276 WatchSource:0}: Error finding container 82f0d37bc9340cce3659cbb568c26976c14a07b273751c7a00a43ba2f2eae276: Status 404 returned error can't find the container with id 82f0d37bc9340cce3659cbb568c26976c14a07b273751c7a00a43ba2f2eae276
Nov 02 01:58:37.656025 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:37.656001    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"]
Nov 02 01:58:38.487152 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:38.487120    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54" event={"ID":"c0d33cf6-0a86-4b65-8c26-bc431f262c3b","Type":"ContainerStarted","Data":"82f0d37bc9340cce3659cbb568c26976c14a07b273751c7a00a43ba2f2eae276"}
Nov 02 01:58:39.541267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:39.541224    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54" event={"ID":"c0d33cf6-0a86-4b65-8c26-bc431f262c3b","Type":"ContainerStarted","Data":"dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9"}
Nov 02 01:58:39.541812 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:39.541301    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:39.671084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:39.667598    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54" podStartSLOduration=2.862259582 podStartE2EDuration="3.66757969s" totalImagesPullingTime="805.320108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:39.66094777 +0000 UTC m=+1612.566413130" watchObservedRunningTime="2026-11-02 01:58:39.66757969 +0000 UTC m=+1612.573045048"
Nov 02 01:58:40.118190 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:40.118160    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:40.302619 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:40.302581    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:40.495412 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:40.495314    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:41.242309 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:41.242277    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:41.557802 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:41.557699    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:42.142931 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:58:42.142892    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:58:42.214768 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:42.214731    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:45.557669 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:45.557643    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 01:58:45.654671 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:45.654640    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:45.840749 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:45.840645    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:46.038413 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.038380    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"]
Nov 02 01:58:46.048391 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.048363    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:46.103477 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.101222    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"]
Nov 02 01:58:46.105853 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.105824    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-25789ee1\""
Nov 02 01:58:46.112699 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.112671    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-25789ee1-dockercfg-qzn56\""
Nov 02 01:58:46.163629 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.163163    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"]
Nov 02 01:58:46.169571 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.169544    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:46.195303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.195267    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7nm4\" (UniqueName: \"kubernetes.io/projected/bb4109c3-77fe-4130-82a7-71a749c94c19-kube-api-access-b7nm4\") pod \"payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp\" (UID: \"bb4109c3-77fe-4130-82a7-71a749c94c19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:46.195600 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.195576    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4109c3-77fe-4130-82a7-71a749c94c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp\" (UID: \"bb4109c3-77fe-4130-82a7-71a749c94c19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:46.219133 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.219102    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"]
Nov 02 01:58:46.297682 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.297640    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4109c3-77fe-4130-82a7-71a749c94c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp\" (UID: \"bb4109c3-77fe-4130-82a7-71a749c94c19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:46.298159 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.298121    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qz7\" (UniqueName: \"kubernetes.io/projected/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-kube-api-access-72qz7\") pod \"payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl\" (UID: \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\") " pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:46.298284 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.298241    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-plugins-config-volume\") pod \"payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl\" (UID: \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\") " pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:46.299267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.298551    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4109c3-77fe-4130-82a7-71a749c94c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp\" (UID: \"bb4109c3-77fe-4130-82a7-71a749c94c19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:46.299267 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.298691    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7nm4\" (UniqueName: \"kubernetes.io/projected/bb4109c3-77fe-4130-82a7-71a749c94c19-kube-api-access-b7nm4\") pod \"payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp\" (UID: \"bb4109c3-77fe-4130-82a7-71a749c94c19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:46.360663 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.360597    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7nm4\" (UniqueName: \"kubernetes.io/projected/bb4109c3-77fe-4130-82a7-71a749c94c19-kube-api-access-b7nm4\") pod \"payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp\" (UID: \"bb4109c3-77fe-4130-82a7-71a749c94c19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:46.365463 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.365440    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:46.402161 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.401521    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qz7\" (UniqueName: \"kubernetes.io/projected/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-kube-api-access-72qz7\") pod \"payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl\" (UID: \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\") " pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:46.402161 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.401639    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-plugins-config-volume\") pod \"payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl\" (UID: \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\") " pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:46.402587 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.402562    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-plugins-config-volume\") pod \"payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl\" (UID: \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\") " pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:46.487773 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.487694    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qz7\" (UniqueName: \"kubernetes.io/projected/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-kube-api-access-72qz7\") pod \"payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl\" (UID: \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\") " pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:46.492013 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.491765    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:46.731977 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.731942    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:46.793294 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:58:46.793262    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4109c3_77fe_4130_82a7_71a749c94c19.slice/crio-e812e71512c61626af898ccad6cad840f86b9b1f7a06c34ead9ecbdd90b2d9eb WatchSource:0}: Error finding container e812e71512c61626af898ccad6cad840f86b9b1f7a06c34ead9ecbdd90b2d9eb: Status 404 returned error can't find the container with id e812e71512c61626af898ccad6cad840f86b9b1f7a06c34ead9ecbdd90b2d9eb
Nov 02 01:58:46.793456 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.793439    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"]
Nov 02 01:58:46.951344 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:58:46.951318    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd162ad4_6a63_49c2_aab7_7e9f74fb5658.slice/crio-2feb7ae697ccf2c525316e8d719e472e026455d6ba34974f13584c26a5c75230 WatchSource:0}: Error finding container 2feb7ae697ccf2c525316e8d719e472e026455d6ba34974f13584c26a5c75230: Status 404 returned error can't find the container with id 2feb7ae697ccf2c525316e8d719e472e026455d6ba34974f13584c26a5c75230
Nov 02 01:58:46.951601 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.951580    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"]
Nov 02 01:58:46.968379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:46.968346    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:47.144444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.144412    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:47.479577 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.479549    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 01:58:47.652455 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.652419    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl" event={"ID":"cd162ad4-6a63-49c2-aab7-7e9f74fb5658","Type":"ContainerStarted","Data":"78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931"}
Nov 02 01:58:47.652652 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.652481    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl" event={"ID":"cd162ad4-6a63-49c2-aab7-7e9f74fb5658","Type":"ContainerStarted","Data":"2feb7ae697ccf2c525316e8d719e472e026455d6ba34974f13584c26a5c75230"}
Nov 02 01:58:47.652714 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.652663    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:47.654375 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.654351    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp" event={"ID":"bb4109c3-77fe-4130-82a7-71a749c94c19","Type":"ContainerStarted","Data":"15425d26e122c52f780aeb54182ce2b5266cdcb440349138268462e0e30f6464"}
Nov 02 01:58:47.654517 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.654386    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp" event={"ID":"bb4109c3-77fe-4130-82a7-71a749c94c19","Type":"ContainerStarted","Data":"e812e71512c61626af898ccad6cad840f86b9b1f7a06c34ead9ecbdd90b2d9eb"}
Nov 02 01:58:47.654517 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.654425    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:47.773119 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.772965    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl" podStartSLOduration=1.772950658 podStartE2EDuration="1.772950658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:47.766764584 +0000 UTC m=+1620.672229944" watchObservedRunningTime="2026-11-02 01:58:47.772950658 +0000 UTC m=+1620.678416013"
Nov 02 01:58:47.836167 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.836133    4912 scope.go:122] "RemoveContainer" containerID="cef463de35f05a3e7d8824f2ba951a3d462599cf971705436d17f174d3978b4f"
Nov 02 01:58:47.936038 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:47.935974    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp" podStartSLOduration=2.93595337 podStartE2EDuration="2.93595337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:47.933626592 +0000 UTC m=+1620.839091951" watchObservedRunningTime="2026-11-02 01:58:47.93595337 +0000 UTC m=+1620.841418730"
Nov 02 01:58:48.092176 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:48.092042    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:50.813356 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:50.813325    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:51.010209 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.010167    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:51.339713 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.339680    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"]
Nov 02 01:58:51.345859 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.345836    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:58:51.423960 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.423913    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"]
Nov 02 01:58:51.460895 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.460850    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-606ca0-dockercfg-xpz5j\""
Nov 02 01:58:51.462262 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.461534    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-606ca0\""
Nov 02 01:58:51.515282 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.515235    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167482e0-1d23-4b89-96af-369f536338e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h\" (UID: \"167482e0-1d23-4b89-96af-369f536338e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:58:51.515506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.515377    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xhg\" (UniqueName: \"kubernetes.io/projected/167482e0-1d23-4b89-96af-369f536338e9-kube-api-access-l6xhg\") pod \"payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h\" (UID: \"167482e0-1d23-4b89-96af-369f536338e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:58:51.617286 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.617248    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167482e0-1d23-4b89-96af-369f536338e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h\" (UID: \"167482e0-1d23-4b89-96af-369f536338e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:58:51.617497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.617368    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xhg\" (UniqueName: \"kubernetes.io/projected/167482e0-1d23-4b89-96af-369f536338e9-kube-api-access-l6xhg\") pod \"payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h\" (UID: \"167482e0-1d23-4b89-96af-369f536338e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:58:51.622044 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.621783    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167482e0-1d23-4b89-96af-369f536338e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h\" (UID: \"167482e0-1d23-4b89-96af-369f536338e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:58:51.679803 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.679618    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xhg\" (UniqueName: \"kubernetes.io/projected/167482e0-1d23-4b89-96af-369f536338e9-kube-api-access-l6xhg\") pod \"payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h\" (UID: \"167482e0-1d23-4b89-96af-369f536338e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:58:51.972795 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:51.972702    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:58:52.369527 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:58:52.369489    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167482e0_1d23_4b89_96af_369f536338e9.slice/crio-97fd7b14dc9d2193e51166cbee551f616271a6828af5fdc65601fd5a271566d8 WatchSource:0}: Error finding container 97fd7b14dc9d2193e51166cbee551f616271a6828af5fdc65601fd5a271566d8: Status 404 returned error can't find the container with id 97fd7b14dc9d2193e51166cbee551f616271a6828af5fdc65601fd5a271566d8
Nov 02 01:58:52.373111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:52.373083    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"]
Nov 02 01:58:52.713909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:52.713809    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h" event={"ID":"167482e0-1d23-4b89-96af-369f536338e9","Type":"ContainerStarted","Data":"03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2"}
Nov 02 01:58:52.713909 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:52.713884    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h" event={"ID":"167482e0-1d23-4b89-96af-369f536338e9","Type":"ContainerStarted","Data":"97fd7b14dc9d2193e51166cbee551f616271a6828af5fdc65601fd5a271566d8"}
Nov 02 01:58:52.714211 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:52.713939    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:58:52.831266 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:52.831215    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h" podStartSLOduration=1.8311975459999998 podStartE2EDuration="1.831197546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:52.830576813 +0000 UTC m=+1625.736042171" watchObservedRunningTime="2026-11-02 01:58:52.831197546 +0000 UTC m=+1625.736662904"
Nov 02 01:58:53.664120 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:53.664081    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:54.932123 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:54.932083    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:55.051349 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:55.051301    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"]
Nov 02 01:58:55.101727 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:55.101693    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f47c3a-5tq9b"]
Nov 02 01:58:55.156314 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:55.156278    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:58:55.229025 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:55.228949    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe0366c-d7e7-4593-84e9-73c34a91c10a" path="/var/lib/kubelet/pods/9fe0366c-d7e7-4593-84e9-73c34a91c10a/volumes"
Nov 02 01:58:58.668291 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:58.668264    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 01:58:58.668787 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:58.668332    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 01:58:58.855726 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:58:58.855688    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:01.053275 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:01.053222    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"]
Nov 02 01:59:01.103370 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:01.103340    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b488eb32-5rqht"]
Nov 02 01:59:01.228742 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:01.228707    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b91b946-671e-48d8-bd8d-f309653460fe" path="/var/lib/kubelet/pods/3b91b946-671e-48d8-bd8d-f309653460fe/volumes"
Nov 02 01:59:03.728379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:03.728351    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 01:59:03.881621 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:03.881586    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:05.049375 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:05.049342    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"]
Nov 02 01:59:05.099256 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:05.099220    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188623e0-g5glw"]
Nov 02 01:59:05.228516 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:05.228485    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bae8290-d9d6-4281-8d86-518aa41e11e1" path="/var/lib/kubelet/pods/4bae8290-d9d6-4281-8d86-518aa41e11e1/volumes"
Nov 02 01:59:07.011763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:07.011726    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:08.639950 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:08.639913    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:12.141346 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:59:12.141301    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:59:13.120643 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:13.120592    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:13.424337 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:13.424257    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:15.049461 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:15.049425    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"]
Nov 02 01:59:15.099660 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:15.099628    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84de8c02-bbxqc"]
Nov 02 01:59:15.228198 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:15.228169    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f88ec0-3de0-459f-bb7a-744218cf3c4d" path="/var/lib/kubelet/pods/53f88ec0-3de0-459f-bb7a-744218cf3c4d/volumes"
Nov 02 01:59:18.154751 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:18.154716    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:23.347591 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:23.347523    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:23.612763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:23.612334    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:23.805382 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:23.805291    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:29.053374 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:29.053298    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"]
Nov 02 01:59:29.105487 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:29.105451    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"]
Nov 02 01:59:29.111742 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:29.111707    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2293eee-ndjtb"]
Nov 02 01:59:29.120365 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:29.120326    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff285d9d-xmrmr"]
Nov 02 01:59:29.230155 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:29.230124    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33581af1-9dd2-4f8a-9bfa-49537e9c652c" path="/var/lib/kubelet/pods/33581af1-9dd2-4f8a-9bfa-49537e9c652c/volumes"
Nov 02 01:59:29.230675 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:29.230661    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c8dc74-fef9-454e-90c8-b844df4403a3" path="/var/lib/kubelet/pods/a1c8dc74-fef9-454e-90c8-b844df4403a3/volumes"
Nov 02 01:59:29.448111 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:29.442778    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:30.269291 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.269256    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"]
Nov 02 01:59:30.275498 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.275472    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.331376 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.331344    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"]
Nov 02 01:59:30.373776 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.373738    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f2f926d-275f7\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.373982 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.373945    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f2f926d-275f7\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.374051 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.374004    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kph87\" (UniqueName: \"kubernetes.io/projected/7463229d-3674-4039-a2f9-ad9a937d1e8c-kube-api-access-kph87\") pod \"maas-api-revoke-keys-e2e-ait-4f2f926d-275f7\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.374497 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.374471    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:59:30.374625 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.374574    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:59:30.475164 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.475132    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f2f926d-275f7\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.475337 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.475187    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kph87\" (UniqueName: \"kubernetes.io/projected/7463229d-3674-4039-a2f9-ad9a937d1e8c-kube-api-access-kph87\") pod \"maas-api-revoke-keys-e2e-ait-4f2f926d-275f7\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.475383 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.475364    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f2f926d-275f7\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.475901 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.475875    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f2f926d-275f7\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.477849 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.477822    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f2f926d-275f7\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.536231 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.536155    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kph87\" (UniqueName: \"kubernetes.io/projected/7463229d-3674-4039-a2f9-ad9a937d1e8c-kube-api-access-kph87\") pod \"maas-api-revoke-keys-e2e-ait-4f2f926d-275f7\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.595474 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.595438    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:30.979845 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:59:30.979802    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7463229d_3674_4039_a2f9_ad9a937d1e8c.slice/crio-c107c6236ead736e7133a48643881ab6bea063cc462cc2a19cff725782441fa7 WatchSource:0}: Error finding container c107c6236ead736e7133a48643881ab6bea063cc462cc2a19cff725782441fa7: Status 404 returned error can't find the container with id c107c6236ead736e7133a48643881ab6bea063cc462cc2a19cff725782441fa7
Nov 02 01:59:30.983000 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:30.982974    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"]
Nov 02 01:59:31.159032 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:31.155182    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7" event={"ID":"7463229d-3674-4039-a2f9-ad9a937d1e8c","Type":"ContainerStarted","Data":"c107c6236ead736e7133a48643881ab6bea063cc462cc2a19cff725782441fa7"}
Nov 02 01:59:32.170455 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:32.170406    4912 generic.go:356] "Generic (PLEG): container finished" podID="7463229d-3674-4039-a2f9-ad9a937d1e8c" containerID="0c887701231d8e61ddfd527b0360d0d8f427678a41a5a781f021fb180dc61129" exitCode=0
Nov 02 01:59:32.170966 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:32.170493    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7" event={"ID":"7463229d-3674-4039-a2f9-ad9a937d1e8c","Type":"ContainerDied","Data":"0c887701231d8e61ddfd527b0360d0d8f427678a41a5a781f021fb180dc61129"}
Nov 02 01:59:33.359662 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.359640    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:33.424517 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.424481    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-xks-ca\") pod \"7463229d-3674-4039-a2f9-ad9a937d1e8c\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") "
Nov 02 01:59:33.424758 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.424744    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7463229d-3674-4039-a2f9-ad9a937d1e8c-kube-api-access-kph87\" (UniqueName: \"kubernetes.io/projected/7463229d-3674-4039-a2f9-ad9a937d1e8c-kube-api-access-kph87\") pod \"7463229d-3674-4039-a2f9-ad9a937d1e8c\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") "
Nov 02 01:59:33.424829 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.424781    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-openshift-ca\") pod \"7463229d-3674-4039-a2f9-ad9a937d1e8c\" (UID: \"7463229d-3674-4039-a2f9-ad9a937d1e8c\") "
Nov 02 01:59:33.425218 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.425191    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-openshift-ca" pod "7463229d-3674-4039-a2f9-ad9a937d1e8c" (UID: "7463229d-3674-4039-a2f9-ad9a937d1e8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:33.425422 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.425405    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:33.427591 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.427561    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7463229d-3674-4039-a2f9-ad9a937d1e8c-kube-api-access-kph87" pod "7463229d-3674-4039-a2f9-ad9a937d1e8c" (UID: "7463229d-3674-4039-a2f9-ad9a937d1e8c"). InnerVolumeSpecName "kube-api-access-kph87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:33.427725 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.427615    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-xks-ca" pod "7463229d-3674-4039-a2f9-ad9a937d1e8c" (UID: "7463229d-3674-4039-a2f9-ad9a937d1e8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:59:33.526569 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.526534    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kph87\" (UniqueName: \"kubernetes.io/projected/7463229d-3674-4039-a2f9-ad9a937d1e8c-kube-api-access-kph87\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:33.526779 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:33.526761    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7463229d-3674-4039-a2f9-ad9a937d1e8c-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:34.050422 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:34.050388    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"]
Nov 02 01:59:34.102815 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:34.102785    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805bec55-jlvvf"]
Nov 02 01:59:34.192670 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:34.192638    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"
Nov 02 01:59:34.192670 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:34.192659    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7" event={"ID":"7463229d-3674-4039-a2f9-ad9a937d1e8c","Type":"ContainerDied","Data":"c107c6236ead736e7133a48643881ab6bea063cc462cc2a19cff725782441fa7"}
Nov 02 01:59:34.192946 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:34.192713    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c107c6236ead736e7133a48643881ab6bea063cc462cc2a19cff725782441fa7"
Nov 02 01:59:34.767990 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:34.767782    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:35.227618 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:35.227581    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d5c846-676b-4baf-94dc-eb1311c002a3" path="/var/lib/kubelet/pods/c0d5c846-676b-4baf-94dc-eb1311c002a3/volumes"
Nov 02 01:59:39.890699 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:39.890665    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"]
Nov 02 01:59:39.891535 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:39.891508    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7463229d-3674-4039-a2f9-ad9a937d1e8c" containerName="revoke-keys"
Nov 02 01:59:39.891596 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:39.891538    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7463229d-3674-4039-a2f9-ad9a937d1e8c" containerName="revoke-keys"
Nov 02 01:59:39.891749 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:39.891738    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="7463229d-3674-4039-a2f9-ad9a937d1e8c" containerName="revoke-keys"
Nov 02 01:59:39.895437 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:39.895420    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:39.954274 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:39.954242    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"]
Nov 02 01:59:39.996701 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:39.996668    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-25789ee1-serving-cert\""
Nov 02 01:59:40.025582 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.025547    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-maas-api-tls\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.025769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.025609    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbcj\" (UniqueName: \"kubernetes.io/projected/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-kube-api-access-tvbcj\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.025769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.025706    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-openshift-service-ca\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.025769 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.025763    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-metrics-certs\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.117386 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.117349    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:40.128365 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.128334    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-maas-api-tls\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.128490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.128432    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbcj\" (UniqueName: \"kubernetes.io/projected/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-kube-api-access-tvbcj\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.128490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.128478    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-openshift-service-ca\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.128576 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.128529    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-metrics-certs\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.130727 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.130145    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-openshift-service-ca\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.141870 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.141785    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-metrics-certs\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.143172 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.143144    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-maas-api-tls\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.191380 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.191348    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbcj\" (UniqueName: \"kubernetes.io/projected/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-kube-api-access-tvbcj\") pod \"maas-api-e2e-mt-25789ee1-6f9c66756d-58998\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.213635 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.213609    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:40.347276 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.347244    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:40.524379 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.524293    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:40.617222 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:59:40.617191    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6a5d2a_07cb_4885_bf16_b26ae2bc6204.slice/crio-9aaf1d2586932b54cc5551b8bc48ad957dac39e72138529dbc6491fc5afd5118 WatchSource:0}: Error finding container 9aaf1d2586932b54cc5551b8bc48ad957dac39e72138529dbc6491fc5afd5118: Status 404 returned error can't find the container with id 9aaf1d2586932b54cc5551b8bc48ad957dac39e72138529dbc6491fc5afd5118
Nov 02 01:59:40.617900 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:40.617881    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"]
Nov 02 01:59:41.278715 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:41.278604    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998" event={"ID":"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204","Type":"ContainerStarted","Data":"9aaf1d2586932b54cc5551b8bc48ad957dac39e72138529dbc6491fc5afd5118"}
Nov 02 01:59:42.142514 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:59:42.142480    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 01:59:42.292592 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:42.292467    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998" event={"ID":"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204","Type":"ContainerStarted","Data":"6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8"}
Nov 02 01:59:42.292592 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:42.292538    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:42.412672 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:42.412559    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998" podStartSLOduration=2.468474295 podStartE2EDuration="3.412539976s" totalImagesPullingTime="944.065681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:42.409168634 +0000 UTC m=+1675.314633992" watchObservedRunningTime="2026-11-02 01:59:42.412539976 +0000 UTC m=+1675.318005336"
Nov 02 01:59:44.052005 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:44.051969    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"]
Nov 02 01:59:44.101645 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:44.101612    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549dde4d-6wtvq"]
Nov 02 01:59:45.227617 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:45.227579    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab59dd4-09e3-4be0-8ea3-843f59f97244" path="/var/lib/kubelet/pods/7ab59dd4-09e3-4be0-8ea3-843f59f97244/volumes"
Nov 02 01:59:45.597529 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:45.597166    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:45.893297 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:45.893260    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"]
Nov 02 01:59:46.007163 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:46.007126    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"]
Nov 02 01:59:46.050827 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:46.050794    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"]
Nov 02 01:59:46.055710 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:46.055689    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0483919d-rg79k"]
Nov 02 01:59:46.281730 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:46.281625    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:46.512882 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:46.512834    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:47.228655 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:47.228622    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f8680e8-d498-4df1-89d2-ab7d69fca0f4" path="/var/lib/kubelet/pods/5f8680e8-d498-4df1-89d2-ab7d69fca0f4/volumes"
Nov 02 01:59:47.300853 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:47.300818    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:47.884763 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:47.884720    4912 scope.go:122] "RemoveContainer" containerID="361b3babb489dc8ca9be78f438c33a52a133aad12b7534cdfd0a34fc389e9a75"
Nov 02 01:59:47.912729 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:47.912703    4912 scope.go:122] "RemoveContainer" containerID="7d7c10823c2828fae62e8548871063ac644e5799d7f9f4768515347d3f99a1a2"
Nov 02 01:59:47.927277 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:47.927256    4912 scope.go:122] "RemoveContainer" containerID="9d00b5ace4902157908328597433c1cb11efd17f79656b0348dfb848e2017b5d"
Nov 02 01:59:48.314539 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:48.314470    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 01:59:49.041716 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:49.041677    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:49.706811 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:49.706753    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:50.056826 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.056727    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"]
Nov 02 01:59:50.114659 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.114625    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a237b1fa-5dkbw"]
Nov 02 01:59:50.141034 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.141000    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:50.813973 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.813938    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"]
Nov 02 01:59:50.820475 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.820451    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:50.877088 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.877038    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"]
Nov 02 01:59:50.894498 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.894437    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66" podUID="b83f3127-e413-4b06-ab1d-a14cf7d65ff4" containerName="payload-processing" containerID="cri-o://a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160" gracePeriod=25
Nov 02 01:59:50.920328 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.920302    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-606ca0-serving-cert\""
Nov 02 01:59:50.937302 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.937275    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwn5c\" (UniqueName: \"kubernetes.io/projected/82976de6-070a-4904-b91c-6f6f0adf8aa7-kube-api-access-vwn5c\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:50.937547 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.937528    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:50.937611 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.937576    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82976de6-070a-4904-b91c-6f6f0adf8aa7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:50.937734 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:50.937718    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.007888 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.007849    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8" podUID="03444ac3-12af-4fa0-b676-56ed9b7d67cb" containerName="payload-pre-processing" containerID="cri-o://a1ecf5489f2285b12a330c519d440a832dd9e94844e3cd0ca681077ffd9221ad" gracePeriod=25
Nov 02 01:59:51.039519 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.039439    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.040303 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.040245    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82976de6-070a-4904-b91c-6f6f0adf8aa7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.040559 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.040382    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.040559 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.040442    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwn5c\" (UniqueName: \"kubernetes.io/projected/82976de6-070a-4904-b91c-6f6f0adf8aa7-kube-api-access-vwn5c\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.040559 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.040493    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:51.040803 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:59:51.040652    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-serving-cert: secret "maas-api-e2e-shared-a-w3-606ca0-serving-cert" not found
Nov 02 01:59:51.040803 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:59:51.040728    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls podName:82976de6-070a-4904-b91c-6f6f0adf8aa7 nodeName:}" failed. No retries permitted until 2026-11-02 01:59:51.540711588 +0000 UTC m=+1684.446176924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls") pod "maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk" (UID: "82976de6-070a-4904-b91c-6f6f0adf8aa7") : secret "maas-api-e2e-shared-a-w3-606ca0-serving-cert" not found
Nov 02 01:59:51.040975 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.040925    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82976de6-070a-4904-b91c-6f6f0adf8aa7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.043003 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.042982    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.115912 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.115632    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwn5c\" (UniqueName: \"kubernetes.io/projected/82976de6-070a-4904-b91c-6f6f0adf8aa7-kube-api-access-vwn5c\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.247885 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.245895    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:51.267520 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.263171    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f02df9b-e86f-4bbc-b93d-624dde3296e0" path="/var/lib/kubelet/pods/4f02df9b-e86f-4bbc-b93d-624dde3296e0/volumes"
Nov 02 01:59:51.282073 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.279406    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:59:51.417233 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.417198    4912 generic.go:356] "Generic (PLEG): container finished" podID="b83f3127-e413-4b06-ab1d-a14cf7d65ff4" containerID="a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160" exitCode=0
Nov 02 01:59:51.417390 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.417260    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"
Nov 02 01:59:51.417390 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.417289    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66" event={"ID":"b83f3127-e413-4b06-ab1d-a14cf7d65ff4","Type":"ContainerDied","Data":"a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160"}
Nov 02 01:59:51.417390 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.417344    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66" event={"ID":"b83f3127-e413-4b06-ab1d-a14cf7d65ff4","Type":"ContainerDied","Data":"51c5aeb31be84b872afe10d67b251ca0178dacec0d06ec70a8c0599204c38dc6"}
Nov 02 01:59:51.417390 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.417377    4912 scope.go:122] "RemoveContainer" containerID="a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160"
Nov 02 01:59:51.419596 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.419571    4912 generic.go:356] "Generic (PLEG): container finished" podID="03444ac3-12af-4fa0-b676-56ed9b7d67cb" containerID="a1ecf5489f2285b12a330c519d440a832dd9e94844e3cd0ca681077ffd9221ad" exitCode=0
Nov 02 01:59:51.419707 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.419646    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8" event={"ID":"03444ac3-12af-4fa0-b676-56ed9b7d67cb","Type":"ContainerDied","Data":"a1ecf5489f2285b12a330c519d440a832dd9e94844e3cd0ca681077ffd9221ad"}
Nov 02 01:59:51.422146 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.422124    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:59:51.430339 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.430316    4912 scope.go:122] "RemoveContainer" containerID="a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160"
Nov 02 01:59:51.430628 ip-10-0-135-33 kubenswrapper[4912]: E1102 01:59:51.430608    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160\": container with ID starting with a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160 not found: ID does not exist" containerID="a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160"
Nov 02 01:59:51.430704 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.430641    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160"} err="failed to get container status \"a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160\": rpc error: code = NotFound desc = could not find container \"a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160\": container with ID starting with a63efd5ccb82542577677814896230c84945645562bd6414df595638898f4160 not found: ID does not exist"
Nov 02 01:59:51.452037 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.452007    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-kube-api-access-fddmq\" (UniqueName: \"kubernetes.io/projected/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-kube-api-access-fddmq\") pod \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\" (UID: \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\") "
Nov 02 01:59:51.452181 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.452141    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03444ac3-12af-4fa0-b676-56ed9b7d67cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03444ac3-12af-4fa0-b676-56ed9b7d67cb-plugins-config-volume\") pod \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\" (UID: \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\") "
Nov 02 01:59:51.452434 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.452413    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03444ac3-12af-4fa0-b676-56ed9b7d67cb-kube-api-access-5mhbk\" (UniqueName: \"kubernetes.io/projected/03444ac3-12af-4fa0-b676-56ed9b7d67cb-kube-api-access-5mhbk\") pod \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\" (UID: \"03444ac3-12af-4fa0-b676-56ed9b7d67cb\") "
Nov 02 01:59:51.452800 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.452521    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03444ac3-12af-4fa0-b676-56ed9b7d67cb-plugins-config-volume" pod "03444ac3-12af-4fa0-b676-56ed9b7d67cb" (UID: "03444ac3-12af-4fa0-b676-56ed9b7d67cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:51.452800 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.452570    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-plugins-config-volume\") pod \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\" (UID: \"b83f3127-e413-4b06-ab1d-a14cf7d65ff4\") "
Nov 02 01:59:51.453098 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.453047    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-plugins-config-volume" pod "b83f3127-e413-4b06-ab1d-a14cf7d65ff4" (UID: "b83f3127-e413-4b06-ab1d-a14cf7d65ff4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:51.453406 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.453386    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03444ac3-12af-4fa0-b676-56ed9b7d67cb-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:51.453490 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.453414    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:51.456090 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.456043    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-kube-api-access-fddmq" pod "b83f3127-e413-4b06-ab1d-a14cf7d65ff4" (UID: "b83f3127-e413-4b06-ab1d-a14cf7d65ff4"). InnerVolumeSpecName "kube-api-access-fddmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:51.456375 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.456343    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03444ac3-12af-4fa0-b676-56ed9b7d67cb-kube-api-access-5mhbk" pod "03444ac3-12af-4fa0-b676-56ed9b7d67cb" (UID: "03444ac3-12af-4fa0-b676-56ed9b7d67cb"). InnerVolumeSpecName "kube-api-access-5mhbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:51.554471 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.554431    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.554713 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.554695    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fddmq\" (UniqueName: \"kubernetes.io/projected/b83f3127-e413-4b06-ab1d-a14cf7d65ff4-kube-api-access-fddmq\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:51.554788 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.554718    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mhbk\" (UniqueName: \"kubernetes.io/projected/03444ac3-12af-4fa0-b676-56ed9b7d67cb-kube-api-access-5mhbk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:51.556983 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.556962    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.737552 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.737523    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:51.847632 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.847600    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"]
Nov 02 01:59:51.852666 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:51.852577    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f2f926d-7c6b8b7b49-zkp66"]
Nov 02 01:59:52.182898 ip-10-0-135-33 kubenswrapper[4912]: W1102 01:59:52.182867    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82976de6_070a_4904_b91c_6f6f0adf8aa7.slice/crio-76da9a5c3ecee3c6f743fe133162e3ce4783d9acd1d19e38b9af8984cb0eb027 WatchSource:0}: Error finding container 76da9a5c3ecee3c6f743fe133162e3ce4783d9acd1d19e38b9af8984cb0eb027: Status 404 returned error can't find the container with id 76da9a5c3ecee3c6f743fe133162e3ce4783d9acd1d19e38b9af8984cb0eb027
Nov 02 01:59:52.184631 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:52.184604    4912 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:59:52.186276 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:52.186090    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"]
Nov 02 01:59:52.431113 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:52.431016    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk" event={"ID":"82976de6-070a-4904-b91c-6f6f0adf8aa7","Type":"ContainerStarted","Data":"76da9a5c3ecee3c6f743fe133162e3ce4783d9acd1d19e38b9af8984cb0eb027"}
Nov 02 01:59:52.432622 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:52.432597    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"
Nov 02 01:59:52.432622 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:52.432615    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8" event={"ID":"03444ac3-12af-4fa0-b676-56ed9b7d67cb","Type":"ContainerDied","Data":"a6e0df8406b292d62a856c3ced3ae587868c3c9e9d1e96fdc9706265b6d0bb06"}
Nov 02 01:59:52.432807 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:52.432662    4912 scope.go:122] "RemoveContainer" containerID="a1ecf5489f2285b12a330c519d440a832dd9e94844e3cd0ca681077ffd9221ad"
Nov 02 01:59:52.600281 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:52.600250    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"]
Nov 02 01:59:52.604875 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:52.604852    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f2f926d-664b5fc9f-2vgj8"]
Nov 02 01:59:53.228937 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:53.228846    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03444ac3-12af-4fa0-b676-56ed9b7d67cb" path="/var/lib/kubelet/pods/03444ac3-12af-4fa0-b676-56ed9b7d67cb/volumes"
Nov 02 01:59:53.229572 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:53.229549    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83f3127-e413-4b06-ab1d-a14cf7d65ff4" path="/var/lib/kubelet/pods/b83f3127-e413-4b06-ab1d-a14cf7d65ff4/volumes"
Nov 02 01:59:53.454801 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:53.454765    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk" event={"ID":"82976de6-070a-4904-b91c-6f6f0adf8aa7","Type":"ContainerStarted","Data":"0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66"}
Nov 02 01:59:53.455016 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:53.454880    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 01:59:53.575416 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:53.575308    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk" podStartSLOduration=2.803755745 podStartE2EDuration="3.575289399s" totalImagesPullingTime="771.533654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:53.571985513 +0000 UTC m=+1686.477450870" watchObservedRunningTime="2026-11-02 01:59:53.575289399 +0000 UTC m=+1686.480754757"
Nov 02 01:59:54.757393 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:54.757357    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:54.948506 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:54.948465    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:55.125364 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:55.125327    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:55.956543 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:55.956507    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"]
Nov 02 01:59:56.069444 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:56.069409    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"]
Nov 02 01:59:56.342930 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:56.342849    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:56.566084 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:56.566019    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:56.751244 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:56.751214    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 01:59:59.479038 ip-10-0-135-33 kubenswrapper[4912]: I1102 01:59:59.479009    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 02:00:00.167774 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.167733    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"]
Nov 02 02:00:00.169010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.168980    4912 cpu_manager.go:414] "containerMap: removing container" podUID="03444ac3-12af-4fa0-b676-56ed9b7d67cb" containerName="payload-pre-processing"
Nov 02 02:00:00.169010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.169007    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03444ac3-12af-4fa0-b676-56ed9b7d67cb" containerName="payload-pre-processing"
Nov 02 02:00:00.169228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.169036    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b83f3127-e413-4b06-ab1d-a14cf7d65ff4" containerName="payload-processing"
Nov 02 02:00:00.169228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.169044    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83f3127-e413-4b06-ab1d-a14cf7d65ff4" containerName="payload-processing"
Nov 02 02:00:00.169290 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.169244    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="03444ac3-12af-4fa0-b676-56ed9b7d67cb" containerName="payload-pre-processing"
Nov 02 02:00:00.169290 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.169257    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83f3127-e413-4b06-ab1d-a14cf7d65ff4" containerName="payload-processing"
Nov 02 02:00:00.174911 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.174891    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"]
Nov 02 02:00:00.175036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.175017    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:00.180299 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.180276    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:00.187802 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.187778    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"]
Nov 02 02:00:00.193237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.193217    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"]
Nov 02 02:00:00.193339 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.193322    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:00.197664 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.197648    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:00.204337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.204311    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtdrl\" (UniqueName: \"kubernetes.io/projected/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-kube-api-access-mtdrl\") pod \"maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q\" (UID: \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:00.204459 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.204410    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa46779-86bb-4ad6-a761-dd4597a027a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2\" (UID: \"8fa46779-86bb-4ad6-a761-dd4597a027a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:00.204548 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.204527    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6adf31e1-c61c-4cef-8302-8bb50cf78730-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr\" (UID: \"6adf31e1-c61c-4cef-8302-8bb50cf78730\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:00.204870 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.204709    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zt8\" (UniqueName: \"kubernetes.io/projected/8fa46779-86bb-4ad6-a761-dd4597a027a8-kube-api-access-f5zt8\") pod \"maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2\" (UID: \"8fa46779-86bb-4ad6-a761-dd4597a027a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:00.205125 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.205104    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmqw\" (UniqueName: \"kubernetes.io/projected/6adf31e1-c61c-4cef-8302-8bb50cf78730-kube-api-access-ghmqw\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr\" (UID: \"6adf31e1-c61c-4cef-8302-8bb50cf78730\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:00.205280 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.205188    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q\" (UID: \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:00.205280 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.205216    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b84770-4707-499e-b149-176a52e064ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt\" (UID: \"78b84770-4707-499e-b149-176a52e064ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:00.205280 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.205249    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frzgv\" (UniqueName: \"kubernetes.io/projected/78b84770-4707-499e-b149-176a52e064ed-kube-api-access-frzgv\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt\" (UID: \"78b84770-4707-499e-b149-176a52e064ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:00.214859 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.214841    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"]
Nov 02 02:00:00.219383 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.219366    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:00.228325 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.228302    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:00:00.267536 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.267497    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"]
Nov 02 02:00:00.272154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.272130    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"]
Nov 02 02:00:00.307168 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.307135    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6adf31e1-c61c-4cef-8302-8bb50cf78730-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr\" (UID: \"6adf31e1-c61c-4cef-8302-8bb50cf78730\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:00.307337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.307229    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zt8\" (UniqueName: \"kubernetes.io/projected/8fa46779-86bb-4ad6-a761-dd4597a027a8-kube-api-access-f5zt8\") pod \"maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2\" (UID: \"8fa46779-86bb-4ad6-a761-dd4597a027a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:00.307337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.307286    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjkn\" (UniqueName: \"kubernetes.io/projected/2e39dce7-46db-440a-8c6b-40d471eee054-kube-api-access-fcjkn\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq\" (UID: \"2e39dce7-46db-440a-8c6b-40d471eee054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:00.307429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.307414    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmqw\" (UniqueName: \"kubernetes.io/projected/6adf31e1-c61c-4cef-8302-8bb50cf78730-kube-api-access-ghmqw\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr\" (UID: \"6adf31e1-c61c-4cef-8302-8bb50cf78730\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:00.307516 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.307500    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q\" (UID: \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:00.307596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.307542    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e39dce7-46db-440a-8c6b-40d471eee054-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq\" (UID: \"2e39dce7-46db-440a-8c6b-40d471eee054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:00.307596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.307570    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b84770-4707-499e-b149-176a52e064ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt\" (UID: \"78b84770-4707-499e-b149-176a52e064ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:00.307704 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.307601    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frzgv\" (UniqueName: \"kubernetes.io/projected/78b84770-4707-499e-b149-176a52e064ed-kube-api-access-frzgv\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt\" (UID: \"78b84770-4707-499e-b149-176a52e064ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:00.307800 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.307776    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtdrl\" (UniqueName: \"kubernetes.io/projected/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-kube-api-access-mtdrl\") pod \"maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q\" (UID: \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:00.308189 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.308107    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa46779-86bb-4ad6-a761-dd4597a027a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2\" (UID: \"8fa46779-86bb-4ad6-a761-dd4597a027a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:00.309115 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.309079    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa46779-86bb-4ad6-a761-dd4597a027a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2\" (UID: \"8fa46779-86bb-4ad6-a761-dd4597a027a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:00.310562 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.310539    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q\" (UID: \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:00.312214 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.311193    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b84770-4707-499e-b149-176a52e064ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt\" (UID: \"78b84770-4707-499e-b149-176a52e064ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:00.319766 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.319741    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6adf31e1-c61c-4cef-8302-8bb50cf78730-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr\" (UID: \"6adf31e1-c61c-4cef-8302-8bb50cf78730\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:00.322522 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.322233    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"]
Nov 02 02:00:00.370821 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.370782    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"]
Nov 02 02:00:00.375399 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.375370    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmqw\" (UniqueName: \"kubernetes.io/projected/6adf31e1-c61c-4cef-8302-8bb50cf78730-kube-api-access-ghmqw\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr\" (UID: \"6adf31e1-c61c-4cef-8302-8bb50cf78730\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:00.379408 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.379384    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"]
Nov 02 02:00:00.380477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.380454    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zt8\" (UniqueName: \"kubernetes.io/projected/8fa46779-86bb-4ad6-a761-dd4597a027a8-kube-api-access-f5zt8\") pod \"maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2\" (UID: \"8fa46779-86bb-4ad6-a761-dd4597a027a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:00.381740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.381719    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtdrl\" (UniqueName: \"kubernetes.io/projected/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-kube-api-access-mtdrl\") pod \"maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q\" (UID: \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:00.382794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.382775    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frzgv\" (UniqueName: \"kubernetes.io/projected/78b84770-4707-499e-b149-176a52e064ed-kube-api-access-frzgv\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt\" (UID: \"78b84770-4707-499e-b149-176a52e064ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:00.409182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.409156    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjkn\" (UniqueName: \"kubernetes.io/projected/2e39dce7-46db-440a-8c6b-40d471eee054-kube-api-access-fcjkn\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq\" (UID: \"2e39dce7-46db-440a-8c6b-40d471eee054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:00.409545 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.409339    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e39dce7-46db-440a-8c6b-40d471eee054-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq\" (UID: \"2e39dce7-46db-440a-8c6b-40d471eee054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:00.410110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.410069    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e39dce7-46db-440a-8c6b-40d471eee054-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq\" (UID: \"2e39dce7-46db-440a-8c6b-40d471eee054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:00.469935 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.469862    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjkn\" (UniqueName: \"kubernetes.io/projected/2e39dce7-46db-440a-8c6b-40d471eee054-kube-api-access-fcjkn\") pod \"maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq\" (UID: \"2e39dce7-46db-440a-8c6b-40d471eee054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:00.494937 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.494903    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:00.500460 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.500435    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:00.514312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.514285    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:00.522730 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.522388    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:00.533562 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.533378    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:00.910001 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:00:00.909974    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b84770_4707_499e_b149_176a52e064ed.slice/crio-da333362ddff3df3edac61956dab460596000d7ed35a5bf2e00081a4917118f0 WatchSource:0}: Error finding container da333362ddff3df3edac61956dab460596000d7ed35a5bf2e00081a4917118f0: Status 404 returned error can't find the container with id da333362ddff3df3edac61956dab460596000d7ed35a5bf2e00081a4917118f0
Nov 02 02:00:00.910521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.910497    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"]
Nov 02 02:00:00.957530 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.957501    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx" podUID="b1588de9-edb4-4d91-8b03-40d02e25c524" containerName="payload-processing" containerID="cri-o://b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78" gracePeriod=25
Nov 02 02:00:00.960295 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:00:00.960262    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa46779_86bb_4ad6_a761_dd4597a027a8.slice/crio-59cff6b7f1aeba41ab4caee9577c8d87c0577459adf48dbad6e9d3c83b24e5fa WatchSource:0}: Error finding container 59cff6b7f1aeba41ab4caee9577c8d87c0577459adf48dbad6e9d3c83b24e5fa: Status 404 returned error can't find the container with id 59cff6b7f1aeba41ab4caee9577c8d87c0577459adf48dbad6e9d3c83b24e5fa
Nov 02 02:00:00.960659 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.960637    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"]
Nov 02 02:00:00.985357 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:00.985333    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"]
Nov 02 02:00:01.011843 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:00:01.011818    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adf31e1_c61c_4cef_8302_8bb50cf78730.slice/crio-6ba880466651d002f43bf65efd7b8c7d472bd7411680d0de9925a03d50b540ba WatchSource:0}: Error finding container 6ba880466651d002f43bf65efd7b8c7d472bd7411680d0de9925a03d50b540ba: Status 404 returned error can't find the container with id 6ba880466651d002f43bf65efd7b8c7d472bd7411680d0de9925a03d50b540ba
Nov 02 02:00:01.012092 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.012045    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"]
Nov 02 02:00:01.020137 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:00:01.020115    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbd6526_94e8_4a43_ad55_b5ec7d71b92e.slice/crio-d9e2fb1b18a8581ed8731679784f8fd05b47cd89dbab86aab8667a4a24ea7124 WatchSource:0}: Error finding container d9e2fb1b18a8581ed8731679784f8fd05b47cd89dbab86aab8667a4a24ea7124: Status 404 returned error can't find the container with id d9e2fb1b18a8581ed8731679784f8fd05b47cd89dbab86aab8667a4a24ea7124
Nov 02 02:00:01.031069 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:00:01.031030    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e39dce7_46db_440a_8c6b_40d471eee054.slice/crio-4fc74d10d62094c1a83612839a1181f63ace4740e56cccfcaa96b1cdd231400d WatchSource:0}: Error finding container 4fc74d10d62094c1a83612839a1181f63ace4740e56cccfcaa96b1cdd231400d: Status 404 returned error can't find the container with id 4fc74d10d62094c1a83612839a1181f63ace4740e56cccfcaa96b1cdd231400d
Nov 02 02:00:01.032024 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.032002    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"]
Nov 02 02:00:01.071701 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.071661    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg" podUID="6b7ca09a-55ac-46fa-9e98-c5b33e3228d4" containerName="payload-pre-processing" containerID="cri-o://8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f" gracePeriod=25
Nov 02 02:00:01.412621 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.412592    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 02:00:01.428747 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.428710    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1588de9-edb4-4d91-8b03-40d02e25c524-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1588de9-edb4-4d91-8b03-40d02e25c524-plugins-config-volume\") pod \"b1588de9-edb4-4d91-8b03-40d02e25c524\" (UID: \"b1588de9-edb4-4d91-8b03-40d02e25c524\") "
Nov 02 02:00:01.428917 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.428805    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1588de9-edb4-4d91-8b03-40d02e25c524-kube-api-access-gdgb2\" (UniqueName: \"kubernetes.io/projected/b1588de9-edb4-4d91-8b03-40d02e25c524-kube-api-access-gdgb2\") pod \"b1588de9-edb4-4d91-8b03-40d02e25c524\" (UID: \"b1588de9-edb4-4d91-8b03-40d02e25c524\") "
Nov 02 02:00:01.430468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.430434    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1588de9-edb4-4d91-8b03-40d02e25c524-plugins-config-volume" pod "b1588de9-edb4-4d91-8b03-40d02e25c524" (UID: "b1588de9-edb4-4d91-8b03-40d02e25c524"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:01.432235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.432207    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1588de9-edb4-4d91-8b03-40d02e25c524-kube-api-access-gdgb2" pod "b1588de9-edb4-4d91-8b03-40d02e25c524" (UID: "b1588de9-edb4-4d91-8b03-40d02e25c524"). InnerVolumeSpecName "kube-api-access-gdgb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:01.531013 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.530980    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1588de9-edb4-4d91-8b03-40d02e25c524-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:01.531013 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.531012    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdgb2\" (UniqueName: \"kubernetes.io/projected/b1588de9-edb4-4d91-8b03-40d02e25c524-kube-api-access-gdgb2\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:01.599500 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.599473    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 02:00:01.632408 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.632380    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-kube-api-access-lrhxt\" (UniqueName: \"kubernetes.io/projected/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-kube-api-access-lrhxt\") pod \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\" (UID: \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\") "
Nov 02 02:00:01.632565 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.632421    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-plugins-config-volume\") pod \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\" (UID: \"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4\") "
Nov 02 02:00:01.632806 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.632777    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-plugins-config-volume" pod "6b7ca09a-55ac-46fa-9e98-c5b33e3228d4" (UID: "6b7ca09a-55ac-46fa-9e98-c5b33e3228d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:01.633032 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.633018    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:01.634837 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.634815    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-kube-api-access-lrhxt" pod "6b7ca09a-55ac-46fa-9e98-c5b33e3228d4" (UID: "6b7ca09a-55ac-46fa-9e98-c5b33e3228d4"). InnerVolumeSpecName "kube-api-access-lrhxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:01.685283 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.685253    4912 generic.go:356] "Generic (PLEG): container finished" podID="6b7ca09a-55ac-46fa-9e98-c5b33e3228d4" containerID="8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f" exitCode=0
Nov 02 02:00:01.685434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.685297    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"
Nov 02 02:00:01.685434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.685331    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg" event={"ID":"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4","Type":"ContainerDied","Data":"8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f"}
Nov 02 02:00:01.685434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.685384    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg" event={"ID":"6b7ca09a-55ac-46fa-9e98-c5b33e3228d4","Type":"ContainerDied","Data":"20c8fff033ab0c61accddd25865c3c8c942e4ee776e62308a97b43b78b698289"}
Nov 02 02:00:01.685434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.685404    4912 scope.go:122] "RemoveContainer" containerID="8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f"
Nov 02 02:00:01.686763 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.686734    4912 generic.go:356] "Generic (PLEG): container finished" podID="b1588de9-edb4-4d91-8b03-40d02e25c524" containerID="b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78" exitCode=0
Nov 02 02:00:01.686862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.686783    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"
Nov 02 02:00:01.686862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.686829    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx" event={"ID":"b1588de9-edb4-4d91-8b03-40d02e25c524","Type":"ContainerDied","Data":"b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78"}
Nov 02 02:00:01.686968 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.686880    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx" event={"ID":"b1588de9-edb4-4d91-8b03-40d02e25c524","Type":"ContainerDied","Data":"08e126caeb7103ac808595f9545b4fa486de9f25b1767e47396c4eb79c1be65e"}
Nov 02 02:00:01.689138 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.689045    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" event={"ID":"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e","Type":"ContainerStarted","Data":"58d12bd950fd5db0330196b58923b4d68fd8c444f39fc7bc09ee1ad371bfe97e"}
Nov 02 02:00:01.689138 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.689107    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" event={"ID":"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e","Type":"ContainerStarted","Data":"d9e2fb1b18a8581ed8731679784f8fd05b47cd89dbab86aab8667a4a24ea7124"}
Nov 02 02:00:01.690990 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.690917    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" event={"ID":"6adf31e1-c61c-4cef-8302-8bb50cf78730","Type":"ContainerStarted","Data":"61e065cc2641caa397e32790f6e482e5907e2d08ed26519011ff7cb46607b688"}
Nov 02 02:00:01.690990 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.690961    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" event={"ID":"6adf31e1-c61c-4cef-8302-8bb50cf78730","Type":"ContainerStarted","Data":"6ba880466651d002f43bf65efd7b8c7d472bd7411680d0de9925a03d50b540ba"}
Nov 02 02:00:01.692821 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.692786    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" event={"ID":"78b84770-4707-499e-b149-176a52e064ed","Type":"ContainerStarted","Data":"c2b34fdcaa791569900ad768c20fbda8bf64661e96927e83571b4112b2c98727"}
Nov 02 02:00:01.692921 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.692835    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" event={"ID":"78b84770-4707-499e-b149-176a52e064ed","Type":"ContainerStarted","Data":"da333362ddff3df3edac61956dab460596000d7ed35a5bf2e00081a4917118f0"}
Nov 02 02:00:01.694095 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.694050    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" event={"ID":"2e39dce7-46db-440a-8c6b-40d471eee054","Type":"ContainerStarted","Data":"4fc74d10d62094c1a83612839a1181f63ace4740e56cccfcaa96b1cdd231400d"}
Nov 02 02:00:01.695925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.695902    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" event={"ID":"8fa46779-86bb-4ad6-a761-dd4597a027a8","Type":"ContainerStarted","Data":"0912d76b23966b0fc534af9bc8c7dd25816ad6ce528be51c9cffda52a8f758de"}
Nov 02 02:00:01.696028 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.695945    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" event={"ID":"8fa46779-86bb-4ad6-a761-dd4597a027a8","Type":"ContainerStarted","Data":"59cff6b7f1aeba41ab4caee9577c8d87c0577459adf48dbad6e9d3c83b24e5fa"}
Nov 02 02:00:01.699864 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.699846    4912 scope.go:122] "RemoveContainer" containerID="8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f"
Nov 02 02:00:01.700140 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:01.700119    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f\": container with ID starting with 8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f not found: ID does not exist" containerID="8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f"
Nov 02 02:00:01.700212 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.700151    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f"} err="failed to get container status \"8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f\": rpc error: code = NotFound desc = could not find container \"8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f\": container with ID starting with 8e1cc703ecb1717bdf2891dc97ec6d6321be0570d1a1af924482a22ca1780e7f not found: ID does not exist"
Nov 02 02:00:01.700212 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.700166    4912 scope.go:122] "RemoveContainer" containerID="b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78"
Nov 02 02:00:01.711023 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.711005    4912 scope.go:122] "RemoveContainer" containerID="b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78"
Nov 02 02:00:01.711277 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:01.711256    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78\": container with ID starting with b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78 not found: ID does not exist" containerID="b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78"
Nov 02 02:00:01.711347 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.711286    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78"} err="failed to get container status \"b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78\": rpc error: code = NotFound desc = could not find container \"b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78\": container with ID starting with b7ec7d4680e85af7803cfdf9351a2ea62a6d02cc6d217963b3799b7ed0d83b78 not found: ID does not exist"
Nov 02 02:00:01.735019 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.734987    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrhxt\" (UniqueName: \"kubernetes.io/projected/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4-kube-api-access-lrhxt\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:01.831237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.831187    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" podStartSLOduration=1.8311696419999999 podStartE2EDuration="1.831169642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:01.829326305 +0000 UTC m=+1694.734791661" watchObservedRunningTime="2026-11-02 02:00:01.831169642 +0000 UTC m=+1694.736634998"
Nov 02 02:00:01.948072 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:01.947969    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" podStartSLOduration=1.947956101 podStartE2EDuration="1.947956101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:01.943004966 +0000 UTC m=+1694.848470321" watchObservedRunningTime="2026-11-02 02:00:01.947956101 +0000 UTC m=+1694.853421456"
Nov 02 02:00:02.062243 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.062187    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" podStartSLOduration=2.062170018 podStartE2EDuration="2.062170018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:02.05623801 +0000 UTC m=+1694.961703366" watchObservedRunningTime="2026-11-02 02:00:02.062170018 +0000 UTC m=+1694.967635377"
Nov 02 02:00:02.172413 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.172385    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"]
Nov 02 02:00:02.177311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.177286    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-af807eae-56c9c4d59-qlktg"]
Nov 02 02:00:02.180259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.180199    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:02.284542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.284497    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"]
Nov 02 02:00:02.292125 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.292090    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-af807eae-7fb64f4789-ktjsx"]
Nov 02 02:00:02.403895 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.403838    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" podStartSLOduration=2.403819139 podStartE2EDuration="2.403819139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:02.395479147 +0000 UTC m=+1695.300944522" watchObservedRunningTime="2026-11-02 02:00:02.403819139 +0000 UTC m=+1695.309284497"
Nov 02 02:00:02.709841 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.709798    4912 generic.go:356] "Generic (PLEG): container finished" podID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerID="58d12bd950fd5db0330196b58923b4d68fd8c444f39fc7bc09ee1ad371bfe97e" exitCode=60
Nov 02 02:00:02.710343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.709865    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" event={"ID":"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e","Type":"ContainerDied","Data":"58d12bd950fd5db0330196b58923b4d68fd8c444f39fc7bc09ee1ad371bfe97e"}
Nov 02 02:00:02.710343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.710251    4912 scope.go:122] "RemoveContainer" containerID="58d12bd950fd5db0330196b58923b4d68fd8c444f39fc7bc09ee1ad371bfe97e"
Nov 02 02:00:02.711728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.711705    4912 generic.go:356] "Generic (PLEG): container finished" podID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerID="61e065cc2641caa397e32790f6e482e5907e2d08ed26519011ff7cb46607b688" exitCode=60
Nov 02 02:00:02.711865 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.711729    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" event={"ID":"6adf31e1-c61c-4cef-8302-8bb50cf78730","Type":"ContainerDied","Data":"61e065cc2641caa397e32790f6e482e5907e2d08ed26519011ff7cb46607b688"}
Nov 02 02:00:02.712136 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.712114    4912 scope.go:122] "RemoveContainer" containerID="61e065cc2641caa397e32790f6e482e5907e2d08ed26519011ff7cb46607b688"
Nov 02 02:00:02.713831 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.713811    4912 generic.go:356] "Generic (PLEG): container finished" podID="78b84770-4707-499e-b149-176a52e064ed" containerID="c2b34fdcaa791569900ad768c20fbda8bf64661e96927e83571b4112b2c98727" exitCode=60
Nov 02 02:00:02.713988 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.713961    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" event={"ID":"78b84770-4707-499e-b149-176a52e064ed","Type":"ContainerDied","Data":"c2b34fdcaa791569900ad768c20fbda8bf64661e96927e83571b4112b2c98727"}
Nov 02 02:00:02.714288 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.714203    4912 scope.go:122] "RemoveContainer" containerID="c2b34fdcaa791569900ad768c20fbda8bf64661e96927e83571b4112b2c98727"
Nov 02 02:00:02.716157 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.716139    4912 generic.go:356] "Generic (PLEG): container finished" podID="2e39dce7-46db-440a-8c6b-40d471eee054" containerID="63fdb7015bca08ed28ab734aad207a32d320877b9b7f3e834778956d2b8139ce" exitCode=60
Nov 02 02:00:02.716255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.716208    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" event={"ID":"2e39dce7-46db-440a-8c6b-40d471eee054","Type":"ContainerDied","Data":"63fdb7015bca08ed28ab734aad207a32d320877b9b7f3e834778956d2b8139ce"}
Nov 02 02:00:02.716465 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.716452    4912 scope.go:122] "RemoveContainer" containerID="63fdb7015bca08ed28ab734aad207a32d320877b9b7f3e834778956d2b8139ce"
Nov 02 02:00:02.718166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.718149    4912 generic.go:356] "Generic (PLEG): container finished" podID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerID="0912d76b23966b0fc534af9bc8c7dd25816ad6ce528be51c9cffda52a8f758de" exitCode=60
Nov 02 02:00:02.718318 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.718305    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" event={"ID":"8fa46779-86bb-4ad6-a761-dd4597a027a8","Type":"ContainerDied","Data":"0912d76b23966b0fc534af9bc8c7dd25816ad6ce528be51c9cffda52a8f758de"}
Nov 02 02:00:02.718570 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:02.718558    4912 scope.go:122] "RemoveContainer" containerID="0912d76b23966b0fc534af9bc8c7dd25816ad6ce528be51c9cffda52a8f758de"
Nov 02 02:00:03.230641 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:03.230568    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7ca09a-55ac-46fa-9e98-c5b33e3228d4" path="/var/lib/kubelet/pods/6b7ca09a-55ac-46fa-9e98-c5b33e3228d4/volumes"
Nov 02 02:00:03.231051 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:03.231037    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1588de9-edb4-4d91-8b03-40d02e25c524" path="/var/lib/kubelet/pods/b1588de9-edb4-4d91-8b03-40d02e25c524/volumes"
Nov 02 02:00:03.729998 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:03.729968    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" event={"ID":"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e","Type":"ContainerStarted","Data":"15dcb9c902873b75071f25dbe9e9cc3b3a4da70b56063921bb0a2a1a9abcf622"}
Nov 02 02:00:03.732024 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:03.731991    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" event={"ID":"6adf31e1-c61c-4cef-8302-8bb50cf78730","Type":"ContainerStarted","Data":"c054009a5ba90ea9a6dedbd8720dd1d5350ccff07784dafa1819d87e4f82e7eb"}
Nov 02 02:00:03.734047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:03.734018    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" event={"ID":"78b84770-4707-499e-b149-176a52e064ed","Type":"ContainerStarted","Data":"10decee78b1c5e1f30015435404020d6c02ba787c43062b22c8f2007d6b4685c"}
Nov 02 02:00:03.736088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:03.736028    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" event={"ID":"2e39dce7-46db-440a-8c6b-40d471eee054","Type":"ContainerStarted","Data":"e3d93bfe454e354d618069220a9defb4d1365ebf0868b538f9fcff390207364d"}
Nov 02 02:00:03.738083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:03.738027    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" event={"ID":"8fa46779-86bb-4ad6-a761-dd4597a027a8","Type":"ContainerStarted","Data":"bcbfdac9149997b57bc463e58fc16d9d26e8684737049cbfa0a1102647115f5e"}
Nov 02 02:00:04.072529 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.072451    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" podStartSLOduration=4.072431027 podStartE2EDuration="4.072431027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:04.072193379 +0000 UTC m=+1696.977658734" watchObservedRunningTime="2026-11-02 02:00:04.072431027 +0000 UTC m=+1696.977896384"
Nov 02 02:00:04.748954 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.748922    4912 generic.go:356] "Generic (PLEG): container finished" podID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerID="15dcb9c902873b75071f25dbe9e9cc3b3a4da70b56063921bb0a2a1a9abcf622" exitCode=60
Nov 02 02:00:04.749500 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.749012    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" event={"ID":"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e","Type":"ContainerDied","Data":"15dcb9c902873b75071f25dbe9e9cc3b3a4da70b56063921bb0a2a1a9abcf622"}
Nov 02 02:00:04.749500 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.749115    4912 scope.go:122] "RemoveContainer" containerID="58d12bd950fd5db0330196b58923b4d68fd8c444f39fc7bc09ee1ad371bfe97e"
Nov 02 02:00:04.749500 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.749315    4912 scope.go:122] "RemoveContainer" containerID="15dcb9c902873b75071f25dbe9e9cc3b3a4da70b56063921bb0a2a1a9abcf622"
Nov 02 02:00:04.749755 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:04.749730    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q_odh-ai-gateway-infra(ccbd6526-94e8-4a43-ad55-b5ec7d71b92e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e"
Nov 02 02:00:04.751342 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.751318    4912 generic.go:356] "Generic (PLEG): container finished" podID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerID="c054009a5ba90ea9a6dedbd8720dd1d5350ccff07784dafa1819d87e4f82e7eb" exitCode=60
Nov 02 02:00:04.751454 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.751398    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" event={"ID":"6adf31e1-c61c-4cef-8302-8bb50cf78730","Type":"ContainerDied","Data":"c054009a5ba90ea9a6dedbd8720dd1d5350ccff07784dafa1819d87e4f82e7eb"}
Nov 02 02:00:04.751637 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.751623    4912 scope.go:122] "RemoveContainer" containerID="c054009a5ba90ea9a6dedbd8720dd1d5350ccff07784dafa1819d87e4f82e7eb"
Nov 02 02:00:04.751991 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:04.751968    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr_odh-ai-gateway-infra(6adf31e1-c61c-4cef-8302-8bb50cf78730)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730"
Nov 02 02:00:04.753472 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.753451    4912 generic.go:356] "Generic (PLEG): container finished" podID="78b84770-4707-499e-b149-176a52e064ed" containerID="10decee78b1c5e1f30015435404020d6c02ba787c43062b22c8f2007d6b4685c" exitCode=60
Nov 02 02:00:04.753568 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.753543    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" event={"ID":"78b84770-4707-499e-b149-176a52e064ed","Type":"ContainerDied","Data":"10decee78b1c5e1f30015435404020d6c02ba787c43062b22c8f2007d6b4685c"}
Nov 02 02:00:04.753775 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.753760    4912 scope.go:122] "RemoveContainer" containerID="10decee78b1c5e1f30015435404020d6c02ba787c43062b22c8f2007d6b4685c"
Nov 02 02:00:04.754183 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:04.754159    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt_odh-ai-gateway-infra(78b84770-4707-499e-b149-176a52e064ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" podUID="78b84770-4707-499e-b149-176a52e064ed"
Nov 02 02:00:04.755656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.755635    4912 generic.go:356] "Generic (PLEG): container finished" podID="2e39dce7-46db-440a-8c6b-40d471eee054" containerID="e3d93bfe454e354d618069220a9defb4d1365ebf0868b538f9fcff390207364d" exitCode=60
Nov 02 02:00:04.755741 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.755685    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" event={"ID":"2e39dce7-46db-440a-8c6b-40d471eee054","Type":"ContainerDied","Data":"e3d93bfe454e354d618069220a9defb4d1365ebf0868b538f9fcff390207364d"}
Nov 02 02:00:04.755969 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.755951    4912 scope.go:122] "RemoveContainer" containerID="e3d93bfe454e354d618069220a9defb4d1365ebf0868b538f9fcff390207364d"
Nov 02 02:00:04.756340 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:04.756318    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq_odh-ai-gateway-infra(2e39dce7-46db-440a-8c6b-40d471eee054)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" podUID="2e39dce7-46db-440a-8c6b-40d471eee054"
Nov 02 02:00:04.757840 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.757820    4912 generic.go:356] "Generic (PLEG): container finished" podID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerID="bcbfdac9149997b57bc463e58fc16d9d26e8684737049cbfa0a1102647115f5e" exitCode=60
Nov 02 02:00:04.757967 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.757945    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" event={"ID":"8fa46779-86bb-4ad6-a761-dd4597a027a8","Type":"ContainerDied","Data":"bcbfdac9149997b57bc463e58fc16d9d26e8684737049cbfa0a1102647115f5e"}
Nov 02 02:00:04.758131 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.758114    4912 scope.go:122] "RemoveContainer" containerID="bcbfdac9149997b57bc463e58fc16d9d26e8684737049cbfa0a1102647115f5e"
Nov 02 02:00:04.758398 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:04.758378    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2_odh-ai-gateway-infra(8fa46779-86bb-4ad6-a761-dd4597a027a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8"
Nov 02 02:00:04.767519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.767502    4912 scope.go:122] "RemoveContainer" containerID="61e065cc2641caa397e32790f6e482e5907e2d08ed26519011ff7cb46607b688"
Nov 02 02:00:04.782749 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.782729    4912 scope.go:122] "RemoveContainer" containerID="c2b34fdcaa791569900ad768c20fbda8bf64661e96927e83571b4112b2c98727"
Nov 02 02:00:04.797612 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.797593    4912 scope.go:122] "RemoveContainer" containerID="63fdb7015bca08ed28ab734aad207a32d320877b9b7f3e834778956d2b8139ce"
Nov 02 02:00:04.812289 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:04.812268    4912 scope.go:122] "RemoveContainer" containerID="0912d76b23966b0fc534af9bc8c7dd25816ad6ce528be51c9cffda52a8f758de"
Nov 02 02:00:05.785385 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:05.785340    4912 scope.go:122] "RemoveContainer" containerID="15dcb9c902873b75071f25dbe9e9cc3b3a4da70b56063921bb0a2a1a9abcf622"
Nov 02 02:00:05.785867 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:05.785783    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q_odh-ai-gateway-infra(ccbd6526-94e8-4a43-ad55-b5ec7d71b92e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e"
Nov 02 02:00:05.787424 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:05.787399    4912 scope.go:122] "RemoveContainer" containerID="c054009a5ba90ea9a6dedbd8720dd1d5350ccff07784dafa1819d87e4f82e7eb"
Nov 02 02:00:05.787818 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:05.787737    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr_odh-ai-gateway-infra(6adf31e1-c61c-4cef-8302-8bb50cf78730)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730"
Nov 02 02:00:05.789487 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:05.789469    4912 scope.go:122] "RemoveContainer" containerID="10decee78b1c5e1f30015435404020d6c02ba787c43062b22c8f2007d6b4685c"
Nov 02 02:00:05.789791 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:05.789763    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt_odh-ai-gateway-infra(78b84770-4707-499e-b149-176a52e064ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" podUID="78b84770-4707-499e-b149-176a52e064ed"
Nov 02 02:00:05.791494 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:05.791474    4912 scope.go:122] "RemoveContainer" containerID="e3d93bfe454e354d618069220a9defb4d1365ebf0868b538f9fcff390207364d"
Nov 02 02:00:05.791829 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:05.791806    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq_odh-ai-gateway-infra(2e39dce7-46db-440a-8c6b-40d471eee054)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" podUID="2e39dce7-46db-440a-8c6b-40d471eee054"
Nov 02 02:00:05.793327 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:05.793309    4912 scope.go:122] "RemoveContainer" containerID="bcbfdac9149997b57bc463e58fc16d9d26e8684737049cbfa0a1102647115f5e"
Nov 02 02:00:05.793626 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:05.793602    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2_odh-ai-gateway-infra(8fa46779-86bb-4ad6-a761-dd4597a027a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8"
Nov 02 02:00:06.049930 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:06.049831    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"]
Nov 02 02:00:06.054686 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:06.054657    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a3713e6-jf7jr"]
Nov 02 02:00:07.230172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:07.230135    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44978b4-76f6-44c2-b83d-ea8c91a209a3" path="/var/lib/kubelet/pods/b44978b4-76f6-44c2-b83d-ea8c91a209a3/volumes"
Nov 02 02:00:07.382615 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:07.382546    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:07.641114 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:07.641069    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:07.870886 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:07.870853    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:08.198932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.198890    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"]
Nov 02 02:00:08.200400 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.200363    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6b7ca09a-55ac-46fa-9e98-c5b33e3228d4" containerName="payload-pre-processing"
Nov 02 02:00:08.200400 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.200392    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7ca09a-55ac-46fa-9e98-c5b33e3228d4" containerName="payload-pre-processing"
Nov 02 02:00:08.200596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.200425    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b1588de9-edb4-4d91-8b03-40d02e25c524" containerName="payload-processing"
Nov 02 02:00:08.200596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.200433    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1588de9-edb4-4d91-8b03-40d02e25c524" containerName="payload-processing"
Nov 02 02:00:08.200719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.200671    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7ca09a-55ac-46fa-9e98-c5b33e3228d4" containerName="payload-pre-processing"
Nov 02 02:00:08.200719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.200687    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1588de9-edb4-4d91-8b03-40d02e25c524" containerName="payload-processing"
Nov 02 02:00:08.209149 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.207412    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:08.250191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.250136    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r89f2\" (UniqueName: \"kubernetes.io/projected/592eef38-f316-4f9c-9a5d-01db442dcf74-kube-api-access-r89f2\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb\" (UID: \"592eef38-f316-4f9c-9a5d-01db442dcf74\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:08.250625 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.250406    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592eef38-f316-4f9c-9a5d-01db442dcf74-plugins-config-volume\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb\" (UID: \"592eef38-f316-4f9c-9a5d-01db442dcf74\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:08.252571 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.252538    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"]
Nov 02 02:00:08.307108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.299256    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5700d3d1-dockercfg-2cf27\""
Nov 02 02:00:08.307108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.299884    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5700d3d1\""
Nov 02 02:00:08.352005 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.351968    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592eef38-f316-4f9c-9a5d-01db442dcf74-plugins-config-volume\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb\" (UID: \"592eef38-f316-4f9c-9a5d-01db442dcf74\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:08.352481 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.352451    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r89f2\" (UniqueName: \"kubernetes.io/projected/592eef38-f316-4f9c-9a5d-01db442dcf74-kube-api-access-r89f2\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb\" (UID: \"592eef38-f316-4f9c-9a5d-01db442dcf74\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:08.356557 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.355372    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592eef38-f316-4f9c-9a5d-01db442dcf74-plugins-config-volume\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb\" (UID: \"592eef38-f316-4f9c-9a5d-01db442dcf74\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:08.395512 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.373865    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"]
Nov 02 02:00:08.395512 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.386981    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:08.414938 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.414907    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r89f2\" (UniqueName: \"kubernetes.io/projected/592eef38-f316-4f9c-9a5d-01db442dcf74-kube-api-access-r89f2\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb\" (UID: \"592eef38-f316-4f9c-9a5d-01db442dcf74\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:08.423462 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.423314    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"]
Nov 02 02:00:08.458690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.457578    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6baaf7a-c642-4575-84a4-7ad1035bcce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z\" (UID: \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:08.458690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.457681    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8bc\" (UniqueName: \"kubernetes.io/projected/a6baaf7a-c642-4575-84a4-7ad1035bcce6-kube-api-access-7p8bc\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z\" (UID: \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:08.536039 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.536004    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:08.559685 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.559655    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6baaf7a-c642-4575-84a4-7ad1035bcce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z\" (UID: \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:08.559881 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.559740    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8bc\" (UniqueName: \"kubernetes.io/projected/a6baaf7a-c642-4575-84a4-7ad1035bcce6-kube-api-access-7p8bc\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z\" (UID: \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:08.560910 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.560467    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6baaf7a-c642-4575-84a4-7ad1035bcce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z\" (UID: \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:08.659334 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.651706    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8bc\" (UniqueName: \"kubernetes.io/projected/a6baaf7a-c642-4575-84a4-7ad1035bcce6-kube-api-access-7p8bc\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z\" (UID: \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:08.727359 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.727279    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:08.955686 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:00:08.955657    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592eef38_f316_4f9c_9a5d_01db442dcf74.slice/crio-b138f926a31bded0c4bc389f90ce3713c10b051db5915c3b314db72cc851b1f6 WatchSource:0}: Error finding container b138f926a31bded0c4bc389f90ce3713c10b051db5915c3b314db72cc851b1f6: Status 404 returned error can't find the container with id b138f926a31bded0c4bc389f90ce3713c10b051db5915c3b314db72cc851b1f6
Nov 02 02:00:08.960545 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:08.960515    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"]
Nov 02 02:00:09.170913 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:00:09.170867    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6baaf7a_c642_4575_84a4_7ad1035bcce6.slice/crio-1ca6ab378b4620714f31de7dab634ddf82c3c4c75e7974f5499d763da18b4de1 WatchSource:0}: Error finding container 1ca6ab378b4620714f31de7dab634ddf82c3c4c75e7974f5499d763da18b4de1: Status 404 returned error can't find the container with id 1ca6ab378b4620714f31de7dab634ddf82c3c4c75e7974f5499d763da18b4de1
Nov 02 02:00:09.175292 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:09.175264    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"]
Nov 02 02:00:09.872367 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:09.872328    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb" event={"ID":"592eef38-f316-4f9c-9a5d-01db442dcf74","Type":"ContainerStarted","Data":"495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7"}
Nov 02 02:00:09.872887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:09.872396    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:09.872887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:09.872422    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb" event={"ID":"592eef38-f316-4f9c-9a5d-01db442dcf74","Type":"ContainerStarted","Data":"b138f926a31bded0c4bc389f90ce3713c10b051db5915c3b314db72cc851b1f6"}
Nov 02 02:00:09.874499 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:09.874467    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z" event={"ID":"a6baaf7a-c642-4575-84a4-7ad1035bcce6","Type":"ContainerStarted","Data":"7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb"}
Nov 02 02:00:09.874639 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:09.874522    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z" event={"ID":"a6baaf7a-c642-4575-84a4-7ad1035bcce6","Type":"ContainerStarted","Data":"1ca6ab378b4620714f31de7dab634ddf82c3c4c75e7974f5499d763da18b4de1"}
Nov 02 02:00:09.874639 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:09.874550    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:09.990770 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:09.990709    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb" podStartSLOduration=1.990689395 podStartE2EDuration="1.990689395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:09.987470174 +0000 UTC m=+1702.892935531" watchObservedRunningTime="2026-11-02 02:00:09.990689395 +0000 UTC m=+1702.896154752"
Nov 02 02:00:12.143951 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:12.143912    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:00:12.444186 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:12.444102    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:14.218925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:14.218894    4912 scope.go:122] "RemoveContainer" containerID="15dcb9c902873b75071f25dbe9e9cc3b3a4da70b56063921bb0a2a1a9abcf622"
Nov 02 02:00:14.924709 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:14.924670    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" event={"ID":"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e","Type":"ContainerStarted","Data":"a8c20526d5e09b086265f3feaf0e0e5ab71571e68d2f9058288ed7577658d9a0"}
Nov 02 02:00:15.040553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:15.040503    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z" podStartSLOduration=7.040490857 podStartE2EDuration="7.040490857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:10.105744836 +0000 UTC m=+1703.011210193" watchObservedRunningTime="2026-11-02 02:00:15.040490857 +0000 UTC m=+1707.945956216"
Nov 02 02:00:15.936253 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:15.936224    4912 generic.go:356] "Generic (PLEG): container finished" podID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerID="a8c20526d5e09b086265f3feaf0e0e5ab71571e68d2f9058288ed7577658d9a0" exitCode=60
Nov 02 02:00:15.936684 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:15.936297    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" event={"ID":"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e","Type":"ContainerDied","Data":"a8c20526d5e09b086265f3feaf0e0e5ab71571e68d2f9058288ed7577658d9a0"}
Nov 02 02:00:15.936684 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:15.936384    4912 scope.go:122] "RemoveContainer" containerID="15dcb9c902873b75071f25dbe9e9cc3b3a4da70b56063921bb0a2a1a9abcf622"
Nov 02 02:00:15.936789 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:15.936741    4912 scope.go:122] "RemoveContainer" containerID="a8c20526d5e09b086265f3feaf0e0e5ab71571e68d2f9058288ed7577658d9a0"
Nov 02 02:00:15.937111 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:15.937046    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q_odh-ai-gateway-infra(ccbd6526-94e8-4a43-ad55-b5ec7d71b92e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e"
Nov 02 02:00:16.051392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:16.051359    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"]
Nov 02 02:00:16.056455 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:16.056407    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"]
Nov 02 02:00:16.061974 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:16.061949    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62c6888-r8kp5"]
Nov 02 02:00:16.218337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:16.218252    4912 scope.go:122] "RemoveContainer" containerID="bcbfdac9149997b57bc463e58fc16d9d26e8684737049cbfa0a1102647115f5e"
Nov 02 02:00:16.218501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:16.218422    4912 scope.go:122] "RemoveContainer" containerID="e3d93bfe454e354d618069220a9defb4d1365ebf0868b538f9fcff390207364d"
Nov 02 02:00:16.950479 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:16.950448    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" event={"ID":"2e39dce7-46db-440a-8c6b-40d471eee054","Type":"ContainerStarted","Data":"813e8b75745232d0fb314cab37264719b7cab5d8327216c8f38790b7febe2f96"}
Nov 02 02:00:16.952995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:16.952969    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" event={"ID":"8fa46779-86bb-4ad6-a761-dd4597a027a8","Type":"ContainerStarted","Data":"dfd8c138b1647b1c53e4fd795673fac1cb3aee183f4263de2bc4c0040d58b8c5"}
Nov 02 02:00:17.137070 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.137031    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:17.218208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.218183    4912 scope.go:122] "RemoveContainer" containerID="c054009a5ba90ea9a6dedbd8720dd1d5350ccff07784dafa1819d87e4f82e7eb"
Nov 02 02:00:17.229324 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.229298    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0939ec83-4504-48a0-9fae-2cc7847ebf16" path="/var/lib/kubelet/pods/0939ec83-4504-48a0-9fae-2cc7847ebf16/volumes"
Nov 02 02:00:17.244257 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.244235    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-kube-api-access-mtdrl\" (UniqueName: \"kubernetes.io/projected/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-kube-api-access-mtdrl\") pod \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\" (UID: \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\") "
Nov 02 02:00:17.244358 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.244276    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-maas-api-service-ca\") pod \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\" (UID: \"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e\") "
Nov 02 02:00:17.244697 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.244657    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-maas-api-service-ca" pod "ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" (UID: "ccbd6526-94e8-4a43-ad55-b5ec7d71b92e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:17.244908 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.244890    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:17.246504 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.246484    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-kube-api-access-mtdrl" pod "ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" (UID: "ccbd6526-94e8-4a43-ad55-b5ec7d71b92e"). InnerVolumeSpecName "kube-api-access-mtdrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:17.347029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.346993    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtdrl\" (UniqueName: \"kubernetes.io/projected/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e-kube-api-access-mtdrl\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:17.453181 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.453024    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:17.964602 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.964574    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"
Nov 02 02:00:17.964602 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.964594    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q" event={"ID":"ccbd6526-94e8-4a43-ad55-b5ec7d71b92e","Type":"ContainerDied","Data":"d9e2fb1b18a8581ed8731679784f8fd05b47cd89dbab86aab8667a4a24ea7124"}
Nov 02 02:00:17.965199 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.964662    4912 scope.go:122] "RemoveContainer" containerID="a8c20526d5e09b086265f3feaf0e0e5ab71571e68d2f9058288ed7577658d9a0"
Nov 02 02:00:17.966827 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.966800    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" event={"ID":"6adf31e1-c61c-4cef-8302-8bb50cf78730","Type":"ContainerStarted","Data":"df690c4d36e9d98a01f8fabf3def72ff071091417c8ba1b0bb1447ddbaa1cdaf"}
Nov 02 02:00:17.969264 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.969235    4912 generic.go:356] "Generic (PLEG): container finished" podID="2e39dce7-46db-440a-8c6b-40d471eee054" containerID="813e8b75745232d0fb314cab37264719b7cab5d8327216c8f38790b7febe2f96" exitCode=60
Nov 02 02:00:17.969730 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.969292    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" event={"ID":"2e39dce7-46db-440a-8c6b-40d471eee054","Type":"ContainerDied","Data":"813e8b75745232d0fb314cab37264719b7cab5d8327216c8f38790b7febe2f96"}
Nov 02 02:00:17.969730 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.969663    4912 scope.go:122] "RemoveContainer" containerID="813e8b75745232d0fb314cab37264719b7cab5d8327216c8f38790b7febe2f96"
Nov 02 02:00:17.970018 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:17.969995    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq_odh-ai-gateway-infra(2e39dce7-46db-440a-8c6b-40d471eee054)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" podUID="2e39dce7-46db-440a-8c6b-40d471eee054"
Nov 02 02:00:17.971590 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.971568    4912 generic.go:356] "Generic (PLEG): container finished" podID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerID="dfd8c138b1647b1c53e4fd795673fac1cb3aee183f4263de2bc4c0040d58b8c5" exitCode=60
Nov 02 02:00:17.971707 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.971640    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" event={"ID":"8fa46779-86bb-4ad6-a761-dd4597a027a8","Type":"ContainerDied","Data":"dfd8c138b1647b1c53e4fd795673fac1cb3aee183f4263de2bc4c0040d58b8c5"}
Nov 02 02:00:17.971991 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.971974    4912 scope.go:122] "RemoveContainer" containerID="dfd8c138b1647b1c53e4fd795673fac1cb3aee183f4263de2bc4c0040d58b8c5"
Nov 02 02:00:17.972272 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:17.972251    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2_odh-ai-gateway-infra(8fa46779-86bb-4ad6-a761-dd4597a027a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8"
Nov 02 02:00:17.979802 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.979789    4912 scope.go:122] "RemoveContainer" containerID="e3d93bfe454e354d618069220a9defb4d1365ebf0868b538f9fcff390207364d"
Nov 02 02:00:17.994357 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:17.994342    4912 scope.go:122] "RemoveContainer" containerID="bcbfdac9149997b57bc463e58fc16d9d26e8684737049cbfa0a1102647115f5e"
Nov 02 02:00:18.132233 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.132194    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"]
Nov 02 02:00:18.137144 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.137118    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-25789ee1-29893080-2pc4q"]
Nov 02 02:00:18.192339 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.192304    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"]
Nov 02 02:00:18.218820 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.218741    4912 scope.go:122] "RemoveContainer" containerID="10decee78b1c5e1f30015435404020d6c02ba787c43062b22c8f2007d6b4685c"
Nov 02 02:00:18.303530 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.303480    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"]
Nov 02 02:00:18.982903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.982866    4912 generic.go:356] "Generic (PLEG): container finished" podID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerID="df690c4d36e9d98a01f8fabf3def72ff071091417c8ba1b0bb1447ddbaa1cdaf" exitCode=60
Nov 02 02:00:18.983388 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.982937    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" event={"ID":"6adf31e1-c61c-4cef-8302-8bb50cf78730","Type":"ContainerDied","Data":"df690c4d36e9d98a01f8fabf3def72ff071091417c8ba1b0bb1447ddbaa1cdaf"}
Nov 02 02:00:18.983388 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.982999    4912 scope.go:122] "RemoveContainer" containerID="c054009a5ba90ea9a6dedbd8720dd1d5350ccff07784dafa1819d87e4f82e7eb"
Nov 02 02:00:18.983509 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.983430    4912 scope.go:122] "RemoveContainer" containerID="df690c4d36e9d98a01f8fabf3def72ff071091417c8ba1b0bb1447ddbaa1cdaf"
Nov 02 02:00:18.983827 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:18.983808    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr_odh-ai-gateway-infra(6adf31e1-c61c-4cef-8302-8bb50cf78730)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730"
Nov 02 02:00:18.985258 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:18.985234    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" event={"ID":"78b84770-4707-499e-b149-176a52e064ed","Type":"ContainerStarted","Data":"8996e08213a4104f7e857e067aa333c61ed4eab0a2b1e9ce793c36b6702260db"}
Nov 02 02:00:19.231078 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.231027    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" path="/var/lib/kubelet/pods/ccbd6526-94e8-4a43-ad55-b5ec7d71b92e/volumes"
Nov 02 02:00:19.231931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.231901    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:19.371259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.371231    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:19.376308 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.376286    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:19.376471 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.376452    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"]
Nov 02 02:00:19.448837 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.448809    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:19.494255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.494223    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa46779-86bb-4ad6-a761-dd4597a027a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa46779-86bb-4ad6-a761-dd4597a027a8-maas-api-service-ca\") pod \"8fa46779-86bb-4ad6-a761-dd4597a027a8\" (UID: \"8fa46779-86bb-4ad6-a761-dd4597a027a8\") "
Nov 02 02:00:19.494437 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.494389    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e39dce7-46db-440a-8c6b-40d471eee054-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e39dce7-46db-440a-8c6b-40d471eee054-maas-api-service-ca\") pod \"2e39dce7-46db-440a-8c6b-40d471eee054\" (UID: \"2e39dce7-46db-440a-8c6b-40d471eee054\") "
Nov 02 02:00:19.494507 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.494494    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e39dce7-46db-440a-8c6b-40d471eee054-kube-api-access-fcjkn\" (UniqueName: \"kubernetes.io/projected/2e39dce7-46db-440a-8c6b-40d471eee054-kube-api-access-fcjkn\") pod \"2e39dce7-46db-440a-8c6b-40d471eee054\" (UID: \"2e39dce7-46db-440a-8c6b-40d471eee054\") "
Nov 02 02:00:19.494566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.494524    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa46779-86bb-4ad6-a761-dd4597a027a8-kube-api-access-f5zt8\" (UniqueName: \"kubernetes.io/projected/8fa46779-86bb-4ad6-a761-dd4597a027a8-kube-api-access-f5zt8\") pod \"8fa46779-86bb-4ad6-a761-dd4597a027a8\" (UID: \"8fa46779-86bb-4ad6-a761-dd4597a027a8\") "
Nov 02 02:00:19.494624 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.494555    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa46779-86bb-4ad6-a761-dd4597a027a8-maas-api-service-ca" pod "8fa46779-86bb-4ad6-a761-dd4597a027a8" (UID: "8fa46779-86bb-4ad6-a761-dd4597a027a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:19.494761 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.494741    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e39dce7-46db-440a-8c6b-40d471eee054-maas-api-service-ca" pod "2e39dce7-46db-440a-8c6b-40d471eee054" (UID: "2e39dce7-46db-440a-8c6b-40d471eee054"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:19.495305 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.495211    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa46779-86bb-4ad6-a761-dd4597a027a8-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:19.495305 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.495234    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e39dce7-46db-440a-8c6b-40d471eee054-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:19.496987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.496963    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa46779-86bb-4ad6-a761-dd4597a027a8-kube-api-access-f5zt8" pod "8fa46779-86bb-4ad6-a761-dd4597a027a8" (UID: "8fa46779-86bb-4ad6-a761-dd4597a027a8"). InnerVolumeSpecName "kube-api-access-f5zt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:19.497101 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.497048    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e39dce7-46db-440a-8c6b-40d471eee054-kube-api-access-fcjkn" pod "2e39dce7-46db-440a-8c6b-40d471eee054" (UID: "2e39dce7-46db-440a-8c6b-40d471eee054"). InnerVolumeSpecName "kube-api-access-fcjkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:19.596402 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.596363    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcjkn\" (UniqueName: \"kubernetes.io/projected/2e39dce7-46db-440a-8c6b-40d471eee054-kube-api-access-fcjkn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:19.596402 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.596394    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5zt8\" (UniqueName: \"kubernetes.io/projected/8fa46779-86bb-4ad6-a761-dd4597a027a8-kube-api-access-f5zt8\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:19.999513 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.999485    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"
Nov 02 02:00:19.999513 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.999502    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq" event={"ID":"2e39dce7-46db-440a-8c6b-40d471eee054","Type":"ContainerDied","Data":"4fc74d10d62094c1a83612839a1181f63ace4740e56cccfcaa96b1cdd231400d"}
Nov 02 02:00:20.000198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:19.999596    4912 scope.go:122] "RemoveContainer" containerID="813e8b75745232d0fb314cab37264719b7cab5d8327216c8f38790b7febe2f96"
Nov 02 02:00:20.009719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.009175    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"
Nov 02 02:00:20.009719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.009233    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2" event={"ID":"8fa46779-86bb-4ad6-a761-dd4597a027a8","Type":"ContainerDied","Data":"59cff6b7f1aeba41ab4caee9577c8d87c0577459adf48dbad6e9d3c83b24e5fa"}
Nov 02 02:00:20.019012 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.018974    4912 scope.go:122] "RemoveContainer" containerID="dfd8c138b1647b1c53e4fd795673fac1cb3aee183f4263de2bc4c0040d58b8c5"
Nov 02 02:00:20.019012 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.018999    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" event={"ID":"78b84770-4707-499e-b149-176a52e064ed","Type":"ContainerDied","Data":"8996e08213a4104f7e857e067aa333c61ed4eab0a2b1e9ce793c36b6702260db"}
Nov 02 02:00:20.019261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.018977    4912 generic.go:356] "Generic (PLEG): container finished" podID="78b84770-4707-499e-b149-176a52e064ed" containerID="8996e08213a4104f7e857e067aa333c61ed4eab0a2b1e9ce793c36b6702260db" exitCode=60
Nov 02 02:00:20.019491 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.019475    4912 scope.go:122] "RemoveContainer" containerID="8996e08213a4104f7e857e067aa333c61ed4eab0a2b1e9ce793c36b6702260db"
Nov 02 02:00:20.019884 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:20.019858    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt_odh-ai-gateway-infra(78b84770-4707-499e-b149-176a52e064ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" podUID="78b84770-4707-499e-b149-176a52e064ed"
Nov 02 02:00:20.034465 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.034443    4912 scope.go:122] "RemoveContainer" containerID="10decee78b1c5e1f30015435404020d6c02ba787c43062b22c8f2007d6b4685c"
Nov 02 02:00:20.174649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.174297    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"]
Nov 02 02:00:20.184113 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.182347    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-af807eae-29893080-kf9l2"]
Nov 02 02:00:20.229531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.229507    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:20.285853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.285762    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"]
Nov 02 02:00:20.297007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.295118    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dbe849-29893080-xq8sq"]
Nov 02 02:00:20.322019 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.321984    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6adf31e1-c61c-4cef-8302-8bb50cf78730-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6adf31e1-c61c-4cef-8302-8bb50cf78730-maas-api-service-ca\") pod \"6adf31e1-c61c-4cef-8302-8bb50cf78730\" (UID: \"6adf31e1-c61c-4cef-8302-8bb50cf78730\") "
Nov 02 02:00:20.322238 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.322199    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adf31e1-c61c-4cef-8302-8bb50cf78730-kube-api-access-ghmqw\" (UniqueName: \"kubernetes.io/projected/6adf31e1-c61c-4cef-8302-8bb50cf78730-kube-api-access-ghmqw\") pod \"6adf31e1-c61c-4cef-8302-8bb50cf78730\" (UID: \"6adf31e1-c61c-4cef-8302-8bb50cf78730\") "
Nov 02 02:00:20.322388 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.322367    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adf31e1-c61c-4cef-8302-8bb50cf78730-maas-api-service-ca" pod "6adf31e1-c61c-4cef-8302-8bb50cf78730" (UID: "6adf31e1-c61c-4cef-8302-8bb50cf78730"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:20.322790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.322770    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6adf31e1-c61c-4cef-8302-8bb50cf78730-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.324550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.324524    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adf31e1-c61c-4cef-8302-8bb50cf78730-kube-api-access-ghmqw" pod "6adf31e1-c61c-4cef-8302-8bb50cf78730" (UID: "6adf31e1-c61c-4cef-8302-8bb50cf78730"). InnerVolumeSpecName "kube-api-access-ghmqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:20.337793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.337770    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"]
Nov 02 02:00:20.423794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.423756    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmqw\" (UniqueName: \"kubernetes.io/projected/6adf31e1-c61c-4cef-8302-8bb50cf78730-kube-api-access-ghmqw\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.888880 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.888845    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:00:20.889077 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:20.888934    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:00:21.036923 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.036878    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr" event={"ID":"6adf31e1-c61c-4cef-8302-8bb50cf78730","Type":"ContainerDied","Data":"6ba880466651d002f43bf65efd7b8c7d472bd7411680d0de9925a03d50b540ba"}
Nov 02 02:00:21.037657 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.036959    4912 scope.go:122] "RemoveContainer" containerID="df690c4d36e9d98a01f8fabf3def72ff071091417c8ba1b0bb1447ddbaa1cdaf"
Nov 02 02:00:21.037657 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.037135    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"
Nov 02 02:00:21.266436 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.266347    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" path="/var/lib/kubelet/pods/2e39dce7-46db-440a-8c6b-40d471eee054/volumes"
Nov 02 02:00:21.266929 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.266912    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" path="/var/lib/kubelet/pods/8fa46779-86bb-4ad6-a761-dd4597a027a8/volumes"
Nov 02 02:00:21.311396 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.311372    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:21.374262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.374232    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"]
Nov 02 02:00:21.379863 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.379838    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893080-7mtfr"]
Nov 02 02:00:21.456438 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.456408    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b84770-4707-499e-b149-176a52e064ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b84770-4707-499e-b149-176a52e064ed-maas-api-service-ca\") pod \"78b84770-4707-499e-b149-176a52e064ed\" (UID: \"78b84770-4707-499e-b149-176a52e064ed\") "
Nov 02 02:00:21.456611 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.456505    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b84770-4707-499e-b149-176a52e064ed-kube-api-access-frzgv\" (UniqueName: \"kubernetes.io/projected/78b84770-4707-499e-b149-176a52e064ed-kube-api-access-frzgv\") pod \"78b84770-4707-499e-b149-176a52e064ed\" (UID: \"78b84770-4707-499e-b149-176a52e064ed\") "
Nov 02 02:00:21.456768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.456747    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b84770-4707-499e-b149-176a52e064ed-maas-api-service-ca" pod "78b84770-4707-499e-b149-176a52e064ed" (UID: "78b84770-4707-499e-b149-176a52e064ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:21.457143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.457123    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b84770-4707-499e-b149-176a52e064ed-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.458822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.458797    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b84770-4707-499e-b149-176a52e064ed-kube-api-access-frzgv" pod "78b84770-4707-499e-b149-176a52e064ed" (UID: "78b84770-4707-499e-b149-176a52e064ed"). InnerVolumeSpecName "kube-api-access-frzgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:21.558210 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:21.558127    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frzgv\" (UniqueName: \"kubernetes.io/projected/78b84770-4707-499e-b149-176a52e064ed-kube-api-access-frzgv\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:22.074273 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:22.074237    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt" event={"ID":"78b84770-4707-499e-b149-176a52e064ed","Type":"ContainerDied","Data":"da333362ddff3df3edac61956dab460596000d7ed35a5bf2e00081a4917118f0"}
Nov 02 02:00:22.074273 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:22.074265    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"
Nov 02 02:00:22.074842 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:22.074310    4912 scope.go:122] "RemoveContainer" containerID="8996e08213a4104f7e857e067aa333c61ed4eab0a2b1e9ce793c36b6702260db"
Nov 02 02:00:22.241761 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:22.241707    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"]
Nov 02 02:00:22.247552 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:22.247517    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893080-2flwt"]
Nov 02 02:00:22.455403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:22.455367    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:23.229562 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:23.229532    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" path="/var/lib/kubelet/pods/6adf31e1-c61c-4cef-8302-8bb50cf78730/volumes"
Nov 02 02:00:23.230038 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:23.230023    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b84770-4707-499e-b149-176a52e064ed" path="/var/lib/kubelet/pods/78b84770-4707-499e-b149-176a52e064ed/volumes"
Nov 02 02:00:26.277682 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:26.277652    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"]
Nov 02 02:00:26.278258 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:26.277917    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy" containerID="cri-o://02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c" gracePeriod=30
Nov 02 02:00:26.567906 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:26.567811    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:28.373888 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:28.373853    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:28.651838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:28.651741    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:28.831522 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:28.831481    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:28.872649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:28.872605    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"]
Nov 02 02:00:28.917688 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:28.917612    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f2f926d-275f7"]
Nov 02 02:00:29.228158 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:29.228049    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7463229d-3674-4039-a2f9-ad9a937d1e8c" path="/var/lib/kubelet/pods/7463229d-3674-4039-a2f9-ad9a937d1e8c/volumes"
Nov 02 02:00:30.370965 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:30.370865    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:31.141179 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.141142    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:31.610238 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.610210    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 02:00:31.658182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.652567    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-podinfo\") pod \"5eac713a-c04b-4a08-9a96-3274f245bb9e\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") "
Nov 02 02:00:31.658182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.652635    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-socket\") pod \"5eac713a-c04b-4a08-9a96-3274f245bb9e\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") "
Nov 02 02:00:31.658182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.652719    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-kube-api-access-2c6kv\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-kube-api-access-2c6kv\") pod \"5eac713a-c04b-4a08-9a96-3274f245bb9e\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") "
Nov 02 02:00:31.658182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.654530    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-socket" pod "5eac713a-c04b-4a08-9a96-3274f245bb9e" (UID: "5eac713a-c04b-4a08-9a96-3274f245bb9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:31.679083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.668251    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-certs\") pod \"5eac713a-c04b-4a08-9a96-3274f245bb9e\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") "
Nov 02 02:00:31.679083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.668453    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-token\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-token\") pod \"5eac713a-c04b-4a08-9a96-3274f245bb9e\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") "
Nov 02 02:00:31.679083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.668483    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-envoy\") pod \"5eac713a-c04b-4a08-9a96-3274f245bb9e\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") "
Nov 02 02:00:31.679083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.668521    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eac713a-c04b-4a08-9a96-3274f245bb9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eac713a-c04b-4a08-9a96-3274f245bb9e-istiod-ca-cert\") pod \"5eac713a-c04b-4a08-9a96-3274f245bb9e\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") "
Nov 02 02:00:31.679083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.668552    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-credential-socket\") pod \"5eac713a-c04b-4a08-9a96-3274f245bb9e\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") "
Nov 02 02:00:31.679083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.668594    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-data\") pod \"5eac713a-c04b-4a08-9a96-3274f245bb9e\" (UID: \"5eac713a-c04b-4a08-9a96-3274f245bb9e\") "
Nov 02 02:00:31.679606 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.679323    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-certs" pod "5eac713a-c04b-4a08-9a96-3274f245bb9e" (UID: "5eac713a-c04b-4a08-9a96-3274f245bb9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:31.689512 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.689459    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-data" pod "5eac713a-c04b-4a08-9a96-3274f245bb9e" (UID: "5eac713a-c04b-4a08-9a96-3274f245bb9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:31.698212 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.698164    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-credential-socket" pod "5eac713a-c04b-4a08-9a96-3274f245bb9e" (UID: "5eac713a-c04b-4a08-9a96-3274f245bb9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:31.706731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.703282    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eac713a-c04b-4a08-9a96-3274f245bb9e-istiod-ca-cert" pod "5eac713a-c04b-4a08-9a96-3274f245bb9e" (UID: "5eac713a-c04b-4a08-9a96-3274f245bb9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:31.706731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.704039    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.706731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.704086    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eac713a-c04b-4a08-9a96-3274f245bb9e-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.706731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.704100    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.706731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.704112    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.706731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.704127    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.718308 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.718271    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-podinfo" pod "5eac713a-c04b-4a08-9a96-3274f245bb9e" (UID: "5eac713a-c04b-4a08-9a96-3274f245bb9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:31.768068 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.729237    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-kube-api-access-2c6kv" pod "5eac713a-c04b-4a08-9a96-3274f245bb9e" (UID: "5eac713a-c04b-4a08-9a96-3274f245bb9e"). InnerVolumeSpecName "kube-api-access-2c6kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:31.768068 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.729335    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-token" pod "5eac713a-c04b-4a08-9a96-3274f245bb9e" (UID: "5eac713a-c04b-4a08-9a96-3274f245bb9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:31.768888 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.768851    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-envoy" pod "5eac713a-c04b-4a08-9a96-3274f245bb9e" (UID: "5eac713a-c04b-4a08-9a96-3274f245bb9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:31.775790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.775621    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:31.817971 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.817932    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.834364 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.834321    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.834613 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.834594    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eac713a-c04b-4a08-9a96-3274f245bb9e-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.834749 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.834731    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c6kv\" (UniqueName: \"kubernetes.io/projected/5eac713a-c04b-4a08-9a96-3274f245bb9e-kube-api-access-2c6kv\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.975231 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:31.975139    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:32.217513 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:32.217479    4912 generic.go:356] "Generic (PLEG): container finished" podID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerID="02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c" exitCode=0
Nov 02 02:00:32.217732 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:32.217621    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" event={"ID":"5eac713a-c04b-4a08-9a96-3274f245bb9e","Type":"ContainerDied","Data":"02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c"}
Nov 02 02:00:32.217732 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:32.217684    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc" event={"ID":"5eac713a-c04b-4a08-9a96-3274f245bb9e","Type":"ContainerDied","Data":"f5c0841433e4c561ed31a9d5539c71e82b7bbef92ec4a6a15cde1eff8532b18a"}
Nov 02 02:00:32.217732 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:32.217705    4912 scope.go:122] "RemoveContainer" containerID="02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c"
Nov 02 02:00:32.217909 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:32.217835    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"
Nov 02 02:00:32.243299 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:32.243273    4912 scope.go:122] "RemoveContainer" containerID="02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c"
Nov 02 02:00:32.243691 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:32.243667    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c\": container with ID starting with 02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c not found: ID does not exist" containerID="02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c"
Nov 02 02:00:32.243793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:32.243702    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c"} err="failed to get container status \"02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c\": rpc error: code = NotFound desc = could not find container \"02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c\": container with ID starting with 02470eda8c1667a03ce9560b3741f3fa6bc6d90d5bd94461152daade506e197c not found: ID does not exist"
Nov 02 02:00:32.391082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:32.390648    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"]
Nov 02 02:00:32.434912 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:32.434880    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-9fzzc"]
Nov 02 02:00:33.230531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:33.230497    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" path="/var/lib/kubelet/pods/5eac713a-c04b-4a08-9a96-3274f245bb9e/volumes"
Nov 02 02:00:34.526946 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:34.526900    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:36.537626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:36.537585    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:36.778977 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:36.778938    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:38.142591 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:38.142526    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:38.319219 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:38.319181    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:39.230894 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.230861    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"]
Nov 02 02:00:39.231449 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.231141    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54" podUID="c0d33cf6-0a86-4b65-8c26-bc431f262c3b" containerName="maas-api" containerID="cri-o://dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9" gracePeriod=30
Nov 02 02:00:39.269080 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.265758    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:39.459459 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.459046    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:39.628142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.628117    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 02:00:39.733654 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.733616    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-kube-api-access-qznkk\" (UniqueName: \"kubernetes.io/projected/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-kube-api-access-qznkk\") pod \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") "
Nov 02 02:00:39.733884 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.733760    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls\") pod \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") "
Nov 02 02:00:39.733884 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.733797    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-metrics-certs\") pod \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") "
Nov 02 02:00:39.733884 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.733856    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-openshift-service-ca\") pod \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\" (UID: \"c0d33cf6-0a86-4b65-8c26-bc431f262c3b\") "
Nov 02 02:00:39.734861 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.734814    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-openshift-service-ca" pod "c0d33cf6-0a86-4b65-8c26-bc431f262c3b" (UID: "c0d33cf6-0a86-4b65-8c26-bc431f262c3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:39.735897 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.735862    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:39.737460 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.737436    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls" pod "c0d33cf6-0a86-4b65-8c26-bc431f262c3b" (UID: "c0d33cf6-0a86-4b65-8c26-bc431f262c3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:00:39.737850 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.737805    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-kube-api-access-qznkk" pod "c0d33cf6-0a86-4b65-8c26-bc431f262c3b" (UID: "c0d33cf6-0a86-4b65-8c26-bc431f262c3b"). InnerVolumeSpecName "kube-api-access-qznkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:39.738144 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.738124    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-metrics-certs" pod "c0d33cf6-0a86-4b65-8c26-bc431f262c3b" (UID: "c0d33cf6-0a86-4b65-8c26-bc431f262c3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:00:39.836257 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.836171    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:39.836257 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.836200    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznkk\" (UniqueName: \"kubernetes.io/projected/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-kube-api-access-qznkk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:39.836257 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.836211    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:39.836257 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:39.836220    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d33cf6-0a86-4b65-8c26-bc431f262c3b-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:40.321372 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.321336    4912 generic.go:356] "Generic (PLEG): container finished" podID="c0d33cf6-0a86-4b65-8c26-bc431f262c3b" containerID="dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9" exitCode=0
Nov 02 02:00:40.323502 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.321408    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"
Nov 02 02:00:40.323502 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.321451    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54" event={"ID":"c0d33cf6-0a86-4b65-8c26-bc431f262c3b","Type":"ContainerDied","Data":"dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9"}
Nov 02 02:00:40.323502 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.321513    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54" event={"ID":"c0d33cf6-0a86-4b65-8c26-bc431f262c3b","Type":"ContainerDied","Data":"82f0d37bc9340cce3659cbb568c26976c14a07b273751c7a00a43ba2f2eae276"}
Nov 02 02:00:40.323502 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.321957    4912 scope.go:122] "RemoveContainer" containerID="dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9"
Nov 02 02:00:40.335469 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.335451    4912 scope.go:122] "RemoveContainer" containerID="dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9"
Nov 02 02:00:40.335801 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:40.335776    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9\": container with ID starting with dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9 not found: ID does not exist" containerID="dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9"
Nov 02 02:00:40.335900 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.335810    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9"} err="failed to get container status \"dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9\": rpc error: code = NotFound desc = could not find container \"dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9\": container with ID starting with dd967db8bc00b13d50943caed306a5ac6520b821c02432d6c0a8b6d1898ab6a9 not found: ID does not exist"
Nov 02 02:00:40.546861 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.546752    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:40.608133 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.601254    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"]
Nov 02 02:00:40.642684 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.642396    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-af807eae-546b45557c-jxf54"]
Nov 02 02:00:40.776014 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.775977    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:40.976582 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:40.976540    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:41.177126 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.176175    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:41.183942 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.183911    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"]
Nov 02 02:00:41.184789 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184769    4912 cpu_manager.go:414] "containerMap: removing container" podUID="c0d33cf6-0a86-4b65-8c26-bc431f262c3b" containerName="maas-api"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184792    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d33cf6-0a86-4b65-8c26-bc431f262c3b" containerName="maas-api"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184813    4912 cpu_manager.go:414] "containerMap: removing container" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerName="cleanup"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184820    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerName="cleanup"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184838    4912 cpu_manager.go:414] "containerMap: removing container" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184846    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184879    4912 cpu_manager.go:414] "containerMap: removing container" podUID="78b84770-4707-499e-b149-176a52e064ed" containerName="cleanup"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184886    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b84770-4707-499e-b149-176a52e064ed" containerName="cleanup"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184901    4912 cpu_manager.go:414] "containerMap: removing container" podUID="78b84770-4707-499e-b149-176a52e064ed" containerName="cleanup"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184908    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b84770-4707-499e-b149-176a52e064ed" containerName="cleanup"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184918    4912 cpu_manager.go:414] "containerMap: removing container" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerName="cleanup"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184925    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerName="cleanup"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184933    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerName="cleanup"
Nov 02 02:00:41.184941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184941    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184951    4912 cpu_manager.go:414] "containerMap: removing container" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184958    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184972    4912 cpu_manager.go:414] "containerMap: removing container" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184979    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184992    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.184999    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185009    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185016    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185035    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185041    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185069    4912 cpu_manager.go:414] "containerMap: removing container" podUID="78b84770-4707-499e-b149-176a52e064ed" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185076    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b84770-4707-499e-b149-176a52e064ed" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185100    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185106    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185135    4912 cpu_manager.go:414] "containerMap: removing container" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185143    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185154    4912 cpu_manager.go:414] "containerMap: removing container" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185160    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185372    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185387    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b84770-4707-499e-b149-176a52e064ed" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185401    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185412    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185425    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185435    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185445    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d33cf6-0a86-4b65-8c26-bc431f262c3b" containerName="maas-api"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185456    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185465    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eac713a-c04b-4a08-9a96-3274f245bb9e" containerName="istio-proxy"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185480    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b84770-4707-499e-b149-176a52e064ed" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185488    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185497    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbd6526-94e8-4a43-ad55-b5ec7d71b92e" containerName="cleanup"
Nov 02 02:00:41.185702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.185505    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerName="cleanup"
Nov 02 02:00:41.192216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.191999    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:41.207540 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.207502    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9nd\" (UniqueName: \"kubernetes.io/projected/8167b26c-f67e-45fc-8691-95a15a9a6d92-kube-api-access-5f9nd\") pod \"payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r\" (UID: \"8167b26c-f67e-45fc-8691-95a15a9a6d92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:41.207699 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.207584    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8167b26c-f67e-45fc-8691-95a15a9a6d92-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r\" (UID: \"8167b26c-f67e-45fc-8691-95a15a9a6d92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:41.234161 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.232529    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d33cf6-0a86-4b65-8c26-bc431f262c3b" path="/var/lib/kubelet/pods/c0d33cf6-0a86-4b65-8c26-bc431f262c3b/volumes"
Nov 02 02:00:41.247807 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.247261    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-129de4\""
Nov 02 02:00:41.248011 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.247884    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-129de4-dockercfg-75vvz\""
Nov 02 02:00:41.248540 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.248518    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"]
Nov 02 02:00:41.309196 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.309160    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9nd\" (UniqueName: \"kubernetes.io/projected/8167b26c-f67e-45fc-8691-95a15a9a6d92-kube-api-access-5f9nd\") pod \"payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r\" (UID: \"8167b26c-f67e-45fc-8691-95a15a9a6d92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:41.309419 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.309235    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8167b26c-f67e-45fc-8691-95a15a9a6d92-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r\" (UID: \"8167b26c-f67e-45fc-8691-95a15a9a6d92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:41.310857 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.310376    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8167b26c-f67e-45fc-8691-95a15a9a6d92-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r\" (UID: \"8167b26c-f67e-45fc-8691-95a15a9a6d92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:41.316450 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.316261    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"]
Nov 02 02:00:41.317434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.317410    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerName="cleanup"
Nov 02 02:00:41.317506 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.317438    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerName="cleanup"
Nov 02 02:00:41.317716 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.317701    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b84770-4707-499e-b149-176a52e064ed" containerName="cleanup"
Nov 02 02:00:41.317780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.317727    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa46779-86bb-4ad6-a761-dd4597a027a8" containerName="cleanup"
Nov 02 02:00:41.317780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.317741    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adf31e1-c61c-4cef-8302-8bb50cf78730" containerName="cleanup"
Nov 02 02:00:41.317780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.317755    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e39dce7-46db-440a-8c6b-40d471eee054" containerName="cleanup"
Nov 02 02:00:41.322880 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.322856    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:41.365876 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.358615    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"]
Nov 02 02:00:41.367187 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.367088    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:41.377106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.376155    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9nd\" (UniqueName: \"kubernetes.io/projected/8167b26c-f67e-45fc-8691-95a15a9a6d92-kube-api-access-5f9nd\") pod \"payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r\" (UID: \"8167b26c-f67e-45fc-8691-95a15a9a6d92\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:41.411497 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.411458    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj5r\" (UniqueName: \"kubernetes.io/projected/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-kube-api-access-blj5r\") pod \"payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr\" (UID: \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:41.411731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.411708    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr\" (UID: \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:41.510269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.510188    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:41.513043 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.513000    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr\" (UID: \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:41.513291 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.513267    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj5r\" (UniqueName: \"kubernetes.io/projected/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-kube-api-access-blj5r\") pod \"payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr\" (UID: \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:41.513893 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.513872    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr\" (UID: \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:41.586706 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.586629    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj5r\" (UniqueName: \"kubernetes.io/projected/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-kube-api-access-blj5r\") pod \"payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr\" (UID: \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:41.718849 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.693363    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:41.724871 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.724835    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:41.969806 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:00:41.969772    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8167b26c_f67e_45fc_8691_95a15a9a6d92.slice/crio-ea677f7f51dc83242be806c54f4a04a46189aebbc50b31910c7d4e80334b9edc WatchSource:0}: Error finding container ea677f7f51dc83242be806c54f4a04a46189aebbc50b31910c7d4e80334b9edc: Status 404 returned error can't find the container with id ea677f7f51dc83242be806c54f4a04a46189aebbc50b31910c7d4e80334b9edc
Nov 02 02:00:41.970462 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:41.970436    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"]
Nov 02 02:00:42.105794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.105755    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:42.143993 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.143854    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"]
Nov 02 02:00:42.149044 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:00:42.149015    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cd8a15_a27f_4c3f_aaa1_f8819bd6584f.slice/crio-329375b87d5375c5f90fff27470f43c259cf64620bf87b948d697eb7c653aead WatchSource:0}: Error finding container 329375b87d5375c5f90fff27470f43c259cf64620bf87b948d697eb7c653aead: Status 404 returned error can't find the container with id 329375b87d5375c5f90fff27470f43c259cf64620bf87b948d697eb7c653aead
Nov 02 02:00:42.176080 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:00:42.169356    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:00:42.288909 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.288859    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:42.385989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.385950    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr" event={"ID":"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f","Type":"ContainerStarted","Data":"c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b"}
Nov 02 02:00:42.386529 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.386019    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr" event={"ID":"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f","Type":"ContainerStarted","Data":"329375b87d5375c5f90fff27470f43c259cf64620bf87b948d697eb7c653aead"}
Nov 02 02:00:42.386529 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.386305    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:42.388215 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.388187    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r" event={"ID":"8167b26c-f67e-45fc-8691-95a15a9a6d92","Type":"ContainerStarted","Data":"96c3c6dbb8d37114a76793cb17a1734fd2dead00ce855a6010d1d16be69907c2"}
Nov 02 02:00:42.388360 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.388237    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r" event={"ID":"8167b26c-f67e-45fc-8691-95a15a9a6d92","Type":"ContainerStarted","Data":"ea677f7f51dc83242be806c54f4a04a46189aebbc50b31910c7d4e80334b9edc"}
Nov 02 02:00:42.388585 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.388565    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:42.507345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.507277    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr" podStartSLOduration=1.507253477 podStartE2EDuration="1.507253477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:42.500812088 +0000 UTC m=+1735.406277448" watchObservedRunningTime="2026-11-02 02:00:42.507253477 +0000 UTC m=+1735.412718835"
Nov 02 02:00:42.625262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.625166    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r" podStartSLOduration=1.6251440069999998 podStartE2EDuration="1.625144007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:42.620615906 +0000 UTC m=+1735.526081266" watchObservedRunningTime="2026-11-02 02:00:42.625144007 +0000 UTC m=+1735.530609364"
Nov 02 02:00:42.959109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:42.958507    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:44.421789 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:44.421755    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:44.771766 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:44.771671    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:47.657566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:47.657526    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:47.943939 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:47.943834    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:48.023997 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.023965    4912 scope.go:122] "RemoveContainer" containerID="fc65840f91daa5642e6b2b1240f2d56c50d5d19675e0a052123082d8acf8bc52"
Nov 02 02:00:48.036140 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.036119    4912 scope.go:122] "RemoveContainer" containerID="2c28938e0e3b02da3f5163df48f04b9927d048a714eb45aa2acdde32ac59a7b2"
Nov 02 02:00:48.047251 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.047228    4912 scope.go:122] "RemoveContainer" containerID="823249985bd088549abae319e572505735ac0b3d0588a95384b2443fbe39ba6c"
Nov 02 02:00:48.060733 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.060709    4912 scope.go:122] "RemoveContainer" containerID="6e322616bfdacb601850bafbcc5efafb9f28441701cfc12a6d0330c02ba89c00"
Nov 02 02:00:48.075455 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.075431    4912 scope.go:122] "RemoveContainer" containerID="1456a90f956c03bcb8bafcb66784ba4f37978f93802b1da3903325392f60fa9b"
Nov 02 02:00:48.087024 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.087000    4912 scope.go:122] "RemoveContainer" containerID="670d6c04552d20fca8912019038e491d9226fac0a7ffdf4a7cfe3b178cb712c8"
Nov 02 02:00:48.099169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.099144    4912 scope.go:122] "RemoveContainer" containerID="64d3dc62ae593491306198c99d98098b67a4e5b46e3b7534670d341fc5475dbe"
Nov 02 02:00:48.111855 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.111670    4912 scope.go:122] "RemoveContainer" containerID="8fd744c58ead86ff01f9e4649f8e99e6dbe256bfe597bdd5f0f250221078c332"
Nov 02 02:00:48.124305 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.124286    4912 scope.go:122] "RemoveContainer" containerID="4b9e40f9913f2ae14dddd5cec93e4aa535e5007f350f14cfb4a99c8c435e3f0f"
Nov 02 02:00:48.136269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.136246    4912 scope.go:122] "RemoveContainer" containerID="64f17248b145f4c7e9385f2dcdcfc28889ef9817f12ef4a3de810854c9861987"
Nov 02 02:00:48.657516 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.657478    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:48.844105 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:48.844026    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:49.029540 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:49.029429    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:49.215280 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:49.215242    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:49.490596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:49.490565    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:49.677825 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:49.677764    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:49.858299 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:49.858219    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:52.963848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:52.963809    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:53.206509 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:53.206463    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:53.409125 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:53.409090    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:00:53.409503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:53.409485    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:00:53.515824 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:53.515790    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:53.701471 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:53.701383    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:53.894527 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:53.894483    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:54.413338 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:54.413300    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:56.217899 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:56.217865    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:56.402485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:56.402452    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:58.232771 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:58.232684    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"]
Nov 02 02:00:58.233295 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:58.232979    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy" containerID="cri-o://c4160f81f7ff2f1358b867e327aa245568f6a323ec178bee9a3f32c68a399215" gracePeriod=30
Nov 02 02:00:58.927879 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:58.927845    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:59.108792 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:59.108730    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:59.292135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:59.287361    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:00:59.577269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:00:59.577181    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:00.242943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:00.242629    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:00.607366 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:00.607262    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:00.928773 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:00.928729    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:01.155029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:01.154993    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:03.642009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:03.641979    4912 generic.go:356] "Generic (PLEG): container finished" podID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerID="c4160f81f7ff2f1358b867e327aa245568f6a323ec178bee9a3f32c68a399215" exitCode=0
Nov 02 02:01:03.642440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:03.642091    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" event={"ID":"25b3f069-354b-490b-b5cf-e002a065f0d6","Type":"ContainerDied","Data":"c4160f81f7ff2f1358b867e327aa245568f6a323ec178bee9a3f32c68a399215"}
Nov 02 02:01:04.029611 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.029584    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.091874    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-envoy\") pod \"25b3f069-354b-490b-b5cf-e002a065f0d6\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") "
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.091941    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-data\") pod \"25b3f069-354b-490b-b5cf-e002a065f0d6\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") "
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.091985    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-kube-api-access-8cht2\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-kube-api-access-8cht2\") pod \"25b3f069-354b-490b-b5cf-e002a065f0d6\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") "
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.092166    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-token\") pod \"25b3f069-354b-490b-b5cf-e002a065f0d6\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") "
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.092296    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b3f069-354b-490b-b5cf-e002a065f0d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f069-354b-490b-b5cf-e002a065f0d6-istiod-ca-cert\") pod \"25b3f069-354b-490b-b5cf-e002a065f0d6\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") "
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.092441    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-podinfo\") pod \"25b3f069-354b-490b-b5cf-e002a065f0d6\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") "
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.092499    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-socket\") pod \"25b3f069-354b-490b-b5cf-e002a065f0d6\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") "
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.092528    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-data" pod "25b3f069-354b-490b-b5cf-e002a065f0d6" (UID: "25b3f069-354b-490b-b5cf-e002a065f0d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.092537    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-certs\") pod \"25b3f069-354b-490b-b5cf-e002a065f0d6\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") "
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.092671    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-credential-socket\") pod \"25b3f069-354b-490b-b5cf-e002a065f0d6\" (UID: \"25b3f069-354b-490b-b5cf-e002a065f0d6\") "
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.093648    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.093876    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-credential-socket" pod "25b3f069-354b-490b-b5cf-e002a065f0d6" (UID: "25b3f069-354b-490b-b5cf-e002a065f0d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.094109    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b3f069-354b-490b-b5cf-e002a065f0d6-istiod-ca-cert" pod "25b3f069-354b-490b-b5cf-e002a065f0d6" (UID: "25b3f069-354b-490b-b5cf-e002a065f0d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:04.094208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.094178    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-socket" pod "25b3f069-354b-490b-b5cf-e002a065f0d6" (UID: "25b3f069-354b-490b-b5cf-e002a065f0d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:04.095051 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.094525    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-envoy" pod "25b3f069-354b-490b-b5cf-e002a065f0d6" (UID: "25b3f069-354b-490b-b5cf-e002a065f0d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:04.095353 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.095322    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-certs" pod "25b3f069-354b-490b-b5cf-e002a065f0d6" (UID: "25b3f069-354b-490b-b5cf-e002a065f0d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:04.096554 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.096505    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-token" pod "25b3f069-354b-490b-b5cf-e002a065f0d6" (UID: "25b3f069-354b-490b-b5cf-e002a065f0d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:04.102049 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.102004    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-kube-api-access-8cht2" pod "25b3f069-354b-490b-b5cf-e002a065f0d6" (UID: "25b3f069-354b-490b-b5cf-e002a065f0d6"). InnerVolumeSpecName "kube-api-access-8cht2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:04.102049 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.102026    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-podinfo" pod "25b3f069-354b-490b-b5cf-e002a065f0d6" (UID: "25b3f069-354b-490b-b5cf-e002a065f0d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:04.197142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.196746    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f069-354b-490b-b5cf-e002a065f0d6-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.197142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.196781    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.197142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.196795    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.197142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.196811    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.197142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.196824    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.197142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.196837    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.197142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.196852    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cht2\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-kube-api-access-8cht2\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.197142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.196867    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f069-354b-490b-b5cf-e002a065f0d6-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.617515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.617478    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:04.684406 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.684371    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"
Nov 02 02:01:04.684406 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.684392    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc" event={"ID":"25b3f069-354b-490b-b5cf-e002a065f0d6","Type":"ContainerDied","Data":"c9da3e34e326c27da3c3cc7d4489a9f45bfc10b2339a01243e76fac7d8667961"}
Nov 02 02:01:04.684995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.684474    4912 scope.go:122] "RemoveContainer" containerID="c4160f81f7ff2f1358b867e327aa245568f6a323ec178bee9a3f32c68a399215"
Nov 02 02:01:04.856174 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.856140    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"]
Nov 02 02:01:04.905092 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:04.904999    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-9rzqc"]
Nov 02 02:01:05.116246 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:05.116207    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:05.230328 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:05.230234    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" path="/var/lib/kubelet/pods/25b3f069-354b-490b-b5cf-e002a065f0d6/volumes"
Nov 02 02:01:05.353149 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:05.353115    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:09.662620 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:09.662581    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:11.052367 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:11.052318    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:12.141378 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:01:12.141341    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:01:14.277786 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:14.277747    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:14.499878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:14.487969    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:14.679139 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:14.678905    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:15.460678 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:15.460637    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"]
Nov 02 02:01:15.570229 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:15.570193    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"]
Nov 02 02:01:15.847358 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:15.847274    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:16.071890 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:16.071836    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:16.990721 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:16.990689    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:19.557965 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:19.557924    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:20.460969 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.460917    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl" podUID="cd162ad4-6a63-49c2-aab7-7e9f74fb5658" containerName="payload-processing" containerID="cri-o://78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931" gracePeriod=25
Nov 02 02:01:20.534331 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:01:20.534287    4912 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd162ad4_6a63_49c2_aab7_7e9f74fb5658.slice/crio-conmon-78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd162ad4_6a63_49c2_aab7_7e9f74fb5658.slice/crio-78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 02:01:20.572684 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.572640    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp" podUID="bb4109c3-77fe-4130-82a7-71a749c94c19" containerName="payload-pre-processing" containerID="cri-o://15425d26e122c52f780aeb54182ce2b5266cdcb440349138268462e0e30f6464" gracePeriod=25
Nov 02 02:01:20.799937 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.799901    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 02:01:20.887445 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.887414    4912 generic.go:356] "Generic (PLEG): container finished" podID="cd162ad4-6a63-49c2-aab7-7e9f74fb5658" containerID="78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931" exitCode=0
Nov 02 02:01:20.887627 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.887459    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"
Nov 02 02:01:20.887627 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.887484    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl" event={"ID":"cd162ad4-6a63-49c2-aab7-7e9f74fb5658","Type":"ContainerDied","Data":"78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931"}
Nov 02 02:01:20.887627 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.887582    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl" event={"ID":"cd162ad4-6a63-49c2-aab7-7e9f74fb5658","Type":"ContainerDied","Data":"2feb7ae697ccf2c525316e8d719e472e026455d6ba34974f13584c26a5c75230"}
Nov 02 02:01:20.887627 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.887607    4912 scope.go:122] "RemoveContainer" containerID="78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931"
Nov 02 02:01:20.888454 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.888435    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 02:01:20.889095 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.889072    4912 generic.go:356] "Generic (PLEG): container finished" podID="bb4109c3-77fe-4130-82a7-71a749c94c19" containerID="15425d26e122c52f780aeb54182ce2b5266cdcb440349138268462e0e30f6464" exitCode=0
Nov 02 02:01:20.889217 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.889108    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp" event={"ID":"bb4109c3-77fe-4130-82a7-71a749c94c19","Type":"ContainerDied","Data":"15425d26e122c52f780aeb54182ce2b5266cdcb440349138268462e0e30f6464"}
Nov 02 02:01:20.899126 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.899105    4912 scope.go:122] "RemoveContainer" containerID="78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931"
Nov 02 02:01:20.899462 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:01:20.899437    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931\": container with ID starting with 78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931 not found: ID does not exist" containerID="78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931"
Nov 02 02:01:20.899541 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.899471    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931"} err="failed to get container status \"78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931\": rpc error: code = NotFound desc = could not find container \"78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931\": container with ID starting with 78cfd074064831d08caa0419196d7b32ca01c10bc84d4e8c4088d157f5791931 not found: ID does not exist"
Nov 02 02:01:20.899541 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.899488    4912 scope.go:122] "RemoveContainer" containerID="15425d26e122c52f780aeb54182ce2b5266cdcb440349138268462e0e30f6464"
Nov 02 02:01:20.916216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.916184    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-plugins-config-volume\") pod \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\" (UID: \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\") "
Nov 02 02:01:20.916376 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.916361    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-kube-api-access-72qz7\" (UniqueName: \"kubernetes.io/projected/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-kube-api-access-72qz7\") pod \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\" (UID: \"cd162ad4-6a63-49c2-aab7-7e9f74fb5658\") "
Nov 02 02:01:20.916814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.916786    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-plugins-config-volume" pod "cd162ad4-6a63-49c2-aab7-7e9f74fb5658" (UID: "cd162ad4-6a63-49c2-aab7-7e9f74fb5658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:20.917605 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.917579    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:20.919167 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:20.919139    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-kube-api-access-72qz7" pod "cd162ad4-6a63-49c2-aab7-7e9f74fb5658" (UID: "cd162ad4-6a63-49c2-aab7-7e9f74fb5658"). InnerVolumeSpecName "kube-api-access-72qz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:21.018436 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.018347    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4109c3-77fe-4130-82a7-71a749c94c19-kube-api-access-b7nm4\" (UniqueName: \"kubernetes.io/projected/bb4109c3-77fe-4130-82a7-71a749c94c19-kube-api-access-b7nm4\") pod \"bb4109c3-77fe-4130-82a7-71a749c94c19\" (UID: \"bb4109c3-77fe-4130-82a7-71a749c94c19\") "
Nov 02 02:01:21.018621 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.018556    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4109c3-77fe-4130-82a7-71a749c94c19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4109c3-77fe-4130-82a7-71a749c94c19-plugins-config-volume\") pod \"bb4109c3-77fe-4130-82a7-71a749c94c19\" (UID: \"bb4109c3-77fe-4130-82a7-71a749c94c19\") "
Nov 02 02:01:21.018961 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.018933    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4109c3-77fe-4130-82a7-71a749c94c19-plugins-config-volume" pod "bb4109c3-77fe-4130-82a7-71a749c94c19" (UID: "bb4109c3-77fe-4130-82a7-71a749c94c19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:21.019103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.019087    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4109c3-77fe-4130-82a7-71a749c94c19-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:21.019350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.019106    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72qz7\" (UniqueName: \"kubernetes.io/projected/cd162ad4-6a63-49c2-aab7-7e9f74fb5658-kube-api-access-72qz7\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:21.020951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.020922    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4109c3-77fe-4130-82a7-71a749c94c19-kube-api-access-b7nm4" pod "bb4109c3-77fe-4130-82a7-71a749c94c19" (UID: "bb4109c3-77fe-4130-82a7-71a749c94c19"). InnerVolumeSpecName "kube-api-access-b7nm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:21.120753 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.120711    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7nm4\" (UniqueName: \"kubernetes.io/projected/bb4109c3-77fe-4130-82a7-71a749c94c19-kube-api-access-b7nm4\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:21.312157 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.312022    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"]
Nov 02 02:01:21.316830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.316798    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25789ee1-bf8c89c79-q4hbl"]
Nov 02 02:01:21.899909 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.899880    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"
Nov 02 02:01:21.899909 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:21.899903    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp" event={"ID":"bb4109c3-77fe-4130-82a7-71a749c94c19","Type":"ContainerDied","Data":"e812e71512c61626af898ccad6cad840f86b9b1f7a06c34ead9ecbdd90b2d9eb"}
Nov 02 02:01:22.065517 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:22.065471    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"]
Nov 02 02:01:22.071867 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:22.071834    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25789ee1-759b5897b7-hpwmp"]
Nov 02 02:01:23.227653 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:23.227618    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4109c3-77fe-4130-82a7-71a749c94c19" path="/var/lib/kubelet/pods/bb4109c3-77fe-4130-82a7-71a749c94c19/volumes"
Nov 02 02:01:23.228120 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:23.228104    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd162ad4-6a63-49c2-aab7-7e9f74fb5658" path="/var/lib/kubelet/pods/cd162ad4-6a63-49c2-aab7-7e9f74fb5658/volumes"
Nov 02 02:01:24.275250 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:24.275216    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:29.160805 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:29.160701    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:33.850506 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:33.850461    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:38.579310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:38.579270    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:38.805255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:38.805157    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:38.991725 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:38.991686    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:42.145165 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:01:42.145122    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:01:43.618376 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:43.618340    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:47.668134 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:47.668107    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:01:47.674832 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:47.674812    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:01:47.731789 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:47.731757    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"]
Nov 02 02:01:47.732297 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:47.732243    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998" podUID="0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" containerName="maas-api" containerID="cri-o://6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8" gracePeriod=30
Nov 02 02:01:47.786016 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:47.785984    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:01:47.798609 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:47.798582    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:01:47.856234 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:47.856191    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:48.100878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.100850    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 02:01:48.128831 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.128793    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:48.152762 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.152728    4912 generic.go:356] "Generic (PLEG): container finished" podID="0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" containerID="6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8" exitCode=0
Nov 02 02:01:48.152955 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.152782    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"
Nov 02 02:01:48.152955 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.152804    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998" event={"ID":"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204","Type":"ContainerDied","Data":"6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8"}
Nov 02 02:01:48.152955 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.152862    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998" event={"ID":"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204","Type":"ContainerDied","Data":"9aaf1d2586932b54cc5551b8bc48ad957dac39e72138529dbc6491fc5afd5118"}
Nov 02 02:01:48.152955 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.152883    4912 scope.go:122] "RemoveContainer" containerID="6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8"
Nov 02 02:01:48.166618 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.166600    4912 scope.go:122] "RemoveContainer" containerID="6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8"
Nov 02 02:01:48.166941 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:01:48.166923    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8\": container with ID starting with 6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8 not found: ID does not exist" containerID="6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8"
Nov 02 02:01:48.167009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.166951    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8"} err="failed to get container status \"6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8\": rpc error: code = NotFound desc = could not find container \"6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8\": container with ID starting with 6773f7f16a2374e8ea87896ae9afe1930239d2c1caa8d49dc1d9dfd7b0d033c8 not found: ID does not exist"
Nov 02 02:01:48.203178 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.203145    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-metrics-certs\") pod \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") "
Nov 02 02:01:48.203361 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.203209    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-maas-api-tls\") pod \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") "
Nov 02 02:01:48.203361 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.203322    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-openshift-service-ca\") pod \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") "
Nov 02 02:01:48.203498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.203362    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-kube-api-access-tvbcj\" (UniqueName: \"kubernetes.io/projected/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-kube-api-access-tvbcj\") pod \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\" (UID: \"0b6a5d2a-07cb-4885-bf16-b26ae2bc6204\") "
Nov 02 02:01:48.203693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.203652    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-openshift-service-ca" pod "0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" (UID: "0b6a5d2a-07cb-4885-bf16-b26ae2bc6204"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:48.204184 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.204136    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:48.205819 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.205790    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-maas-api-tls" pod "0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" (UID: "0b6a5d2a-07cb-4885-bf16-b26ae2bc6204"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:48.205819 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.205809    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-kube-api-access-tvbcj" pod "0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" (UID: "0b6a5d2a-07cb-4885-bf16-b26ae2bc6204"). InnerVolumeSpecName "kube-api-access-tvbcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:48.205957 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.205809    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-metrics-certs" pod "0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" (UID: "0b6a5d2a-07cb-4885-bf16-b26ae2bc6204"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:48.305436 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.305390    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvbcj\" (UniqueName: \"kubernetes.io/projected/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-kube-api-access-tvbcj\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:48.305436 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.305427    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:48.305436 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.305441    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:48.329959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.329876    4912 scope.go:122] "RemoveContainer" containerID="df8e2f95b3045aadae0301f85de201fdb7ab9464f29953e83ac68df389764fa2"
Nov 02 02:01:48.341730 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.341708    4912 scope.go:122] "RemoveContainer" containerID="152691f186fe62cbb7c5cf46fb048bc0484717f88239751252003aff292b055a"
Nov 02 02:01:48.351693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.351671    4912 scope.go:122] "RemoveContainer" containerID="f397a7478deb559853894faab5f1d8977f6fa4b24c456a66a258a880b55e48fe"
Nov 02 02:01:48.597769 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.597692    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"]
Nov 02 02:01:48.604978 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:48.604952    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25789ee1-6f9c66756d-58998"]
Nov 02 02:01:49.230171 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:49.230135    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" path="/var/lib/kubelet/pods/0b6a5d2a-07cb-4885-bf16-b26ae2bc6204/volumes"
Nov 02 02:01:49.451461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:49.451416    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:49.790914 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:49.790864    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:49.976669 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:49.976634    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:50.414235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:50.414193    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:52.615338 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:52.615302    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:55.130586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:55.130551    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:56.635249 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:56.635216    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:56.814179 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:56.814128    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:57.060010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:57.059814    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:58.208221 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:58.208178    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:59.087681 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:59.087644    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:01:59.295544 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:01:59.295508    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:00.351753 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:00.351717    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:03.558083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:03.554655    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:03.748223 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:03.748172    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:03.948183 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:03.948141    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:05.523768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:05.523732    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:06.816544 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:06.816510    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:07.000033 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:06.999999    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:10.880568 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:10.880529    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:12.140589 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:02:12.140548    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:02:16.084013 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:16.083976    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:16.305855 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:16.305818    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:17.925338 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:17.925305    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:21.353725 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:21.353685    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:22.167426 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:22.167232    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:26.507645 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:26.507611    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:31.684239 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:31.684133    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:36.649736 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:36.649679    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:41.458108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:41.458029    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:41.653528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:41.653492    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:42.331081 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:02:42.323959    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:02:42.371084 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:42.368461    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:42.807408 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:42.807371    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:45.379141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.379102    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"]
Nov 02 02:02:45.379903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.379886    4912 cpu_manager.go:414] "containerMap: removing container" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy"
Nov 02 02:02:45.379956 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.379905    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy"
Nov 02 02:02:45.379956 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.379926    4912 cpu_manager.go:414] "containerMap: removing container" podUID="0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" containerName="maas-api"
Nov 02 02:02:45.379956 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.379931    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" containerName="maas-api"
Nov 02 02:02:45.380094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.379970    4912 cpu_manager.go:414] "containerMap: removing container" podUID="bb4109c3-77fe-4130-82a7-71a749c94c19" containerName="payload-pre-processing"
Nov 02 02:02:45.380094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.379977    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4109c3-77fe-4130-82a7-71a749c94c19" containerName="payload-pre-processing"
Nov 02 02:02:45.380094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.379988    4912 cpu_manager.go:414] "containerMap: removing container" podUID="cd162ad4-6a63-49c2-aab7-7e9f74fb5658" containerName="payload-processing"
Nov 02 02:02:45.380094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.379993    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd162ad4-6a63-49c2-aab7-7e9f74fb5658" containerName="payload-processing"
Nov 02 02:02:45.380228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.380117    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b3f069-354b-490b-b5cf-e002a065f0d6" containerName="istio-proxy"
Nov 02 02:02:45.380228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.380127    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6a5d2a-07cb-4885-bf16-b26ae2bc6204" containerName="maas-api"
Nov 02 02:02:45.380228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.380140    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4109c3-77fe-4130-82a7-71a749c94c19" containerName="payload-pre-processing"
Nov 02 02:02:45.380228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.380145    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd162ad4-6a63-49c2-aab7-7e9f74fb5658" containerName="payload-processing"
Nov 02 02:02:45.383932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.383912    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.429506 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.429459    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-istio-token\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.429698 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.429633    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.429765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.429727    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-credential-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.429813 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.429796    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-certs\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.430047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.430020    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-envoy\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.430159 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.430096    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxw4x\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-kube-api-access-kxw4x\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.430159 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.430137    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363846bf-0e6d-466e-b458-97597f12d480-istiod-ca-cert\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.430272 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.430230    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363846bf-0e6d-466e-b458-97597f12d480-istio-podinfo\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.430272 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.430258    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-data\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.491838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.491800    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"]
Nov 02 02:02:45.519815 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.519782    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c1aad5-openshift-default-dockercfg-8lcpp\""
Nov 02 02:02:45.531821 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.531791    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.532011 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.531848    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-credential-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.532011 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.531898    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-certs\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.532174 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.532049    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-envoy\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.532174 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.532090    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxw4x\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-kube-api-access-kxw4x\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.532174 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.532117    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363846bf-0e6d-466e-b458-97597f12d480-istiod-ca-cert\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.532174 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.532149    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363846bf-0e6d-466e-b458-97597f12d480-istio-podinfo\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.532389 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.532177    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-data\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.532389 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.532210    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-istio-token\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.532913 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.532880    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.533429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.533401    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-data\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.533532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.533429    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363846bf-0e6d-466e-b458-97597f12d480-istiod-ca-cert\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.533651 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.533618    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-credential-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.533862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.533831    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-certs\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.536135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.536116    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363846bf-0e6d-466e-b458-97597f12d480-istio-podinfo\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.536414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.536394    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-envoy\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.594019 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.593990    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxw4x\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-kube-api-access-kxw4x\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.594183 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.593994    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-istio-token\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:45.707183 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:45.707094    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:46.150596 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:02:46.150559    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363846bf_0e6d_466e_b458_97597f12d480.slice/crio-761933e4a2549bc8819dd3fab92b4d01131d8a77aab5ce8857ecf7e0d100cf1d WatchSource:0}: Error finding container 761933e4a2549bc8819dd3fab92b4d01131d8a77aab5ce8857ecf7e0d100cf1d: Status 404 returned error can't find the container with id 761933e4a2549bc8819dd3fab92b4d01131d8a77aab5ce8857ecf7e0d100cf1d
Nov 02 02:02:46.150803 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:46.150782    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"]
Nov 02 02:02:46.153272 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:46.153242    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:02:46.153369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:46.153303    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:02:46.153369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:46.153331    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:02:46.581627 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:46.581533    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:46.762480 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:46.762443    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" event={"ID":"363846bf-0e6d-466e-b458-97597f12d480","Type":"ContainerStarted","Data":"58cdae4d2708055b31e394ca340212be792c95420df3eebe073f630f460a2164"}
Nov 02 02:02:46.762672 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:46.762502    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" event={"ID":"363846bf-0e6d-466e-b458-97597f12d480","Type":"ContainerStarted","Data":"761933e4a2549bc8819dd3fab92b4d01131d8a77aab5ce8857ecf7e0d100cf1d"}
Nov 02 02:02:46.884732 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:46.884672    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" podStartSLOduration=1.884657131 podStartE2EDuration="1.884657131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:46.881343021 +0000 UTC m=+1859.786808379" watchObservedRunningTime="2026-11-02 02:02:46.884657131 +0000 UTC m=+1859.790122487"
Nov 02 02:02:47.285763 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:47.285673    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:47.482619 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:47.482583    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:47.678582 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:47.678543    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:47.707509 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:47.707476    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:47.709115 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:47.709072    4912 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 02:02:47.709259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:47.709122    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 02:02:48.386863 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:48.386822    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:48.707480 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:48.707391    4912 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 02:02:48.707480 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:48.707450    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 02:02:49.707871 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:49.707809    4912 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 02:02:49.708411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:49.707887    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 02:02:50.514804 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:50.514771    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:50.755772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:50.726760    4912 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 02:02:50.755772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:50.726888    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 02:02:50.834144 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:50.833386    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:51.708340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:51.708305    4912 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 02:02:51.708571 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:51.708375    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 02:02:51.976824 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:51.976741    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:02:52.707660 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:52.707629    4912 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 02:02:52.707946 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:52.707910    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 02:02:53.712304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:53.712267    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:53.712696 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:53.712658    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:53.713524 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:53.713506    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:02:56.989166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:02:56.989125    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:02.333715 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:02.333676    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:04.615148 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:04.615091    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:05.033080 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:05.032969    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:05.797085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:05.793555    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:06.270463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:06.270418    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:07.639133 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:07.639098    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:10.472180 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:10.472141    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:10.709727 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:10.709688    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:12.286514 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:03:12.286475    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:03:12.668925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:12.668889    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"]
Nov 02 02:03:12.669445 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:12.669404    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy" containerID="cri-o://0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec" gracePeriod=30
Nov 02 02:03:13.103407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:13.103325    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:16.042182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:16.042144    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:16.229479 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:16.229443    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:16.411232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:16.411111    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:18.095233 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.095201    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 02:03:18.107646 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.107620    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-envoy\") pod \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") "
Nov 02 02:03:18.107839 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.107779    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-certs\") pod \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") "
Nov 02 02:03:18.107839 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.107813    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-podinfo\") pod \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") "
Nov 02 02:03:18.107839 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.107833    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-data\") pod \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") "
Nov 02 02:03:18.108025 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.107862    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-token\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-token\") pod \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") "
Nov 02 02:03:18.108025 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.108017    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-credential-socket\") pod \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") "
Nov 02 02:03:18.108170 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.108089    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-certs" pod "31c4c212-d4ce-4b23-be07-a85fb4ded4f7" (UID: "31c4c212-d4ce-4b23-be07-a85fb4ded4f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.108170 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.108122    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istiod-ca-cert\") pod \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") "
Nov 02 02:03:18.108242 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.108196    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-kube-api-access-nj6j9\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-kube-api-access-nj6j9\") pod \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") "
Nov 02 02:03:18.108300 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.108237    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-socket\") pod \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\" (UID: \"31c4c212-d4ce-4b23-be07-a85fb4ded4f7\") "
Nov 02 02:03:18.108342 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.108318    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-data" pod "31c4c212-d4ce-4b23-be07-a85fb4ded4f7" (UID: "31c4c212-d4ce-4b23-be07-a85fb4ded4f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.108742 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.108689    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istiod-ca-cert" pod "31c4c212-d4ce-4b23-be07-a85fb4ded4f7" (UID: "31c4c212-d4ce-4b23-be07-a85fb4ded4f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:18.109312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.109148    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-socket" pod "31c4c212-d4ce-4b23-be07-a85fb4ded4f7" (UID: "31c4c212-d4ce-4b23-be07-a85fb4ded4f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.109520 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.109469    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-credential-socket" pod "31c4c212-d4ce-4b23-be07-a85fb4ded4f7" (UID: "31c4c212-d4ce-4b23-be07-a85fb4ded4f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.109648 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.109527    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.109648 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.109566    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.109648 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.109581    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.109648 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.109595    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.110714 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.110695    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-envoy" pod "31c4c212-d4ce-4b23-be07-a85fb4ded4f7" (UID: "31c4c212-d4ce-4b23-be07-a85fb4ded4f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.110863 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.110842    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-podinfo" pod "31c4c212-d4ce-4b23-be07-a85fb4ded4f7" (UID: "31c4c212-d4ce-4b23-be07-a85fb4ded4f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:03:18.110932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.110904    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-token" pod "31c4c212-d4ce-4b23-be07-a85fb4ded4f7" (UID: "31c4c212-d4ce-4b23-be07-a85fb4ded4f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:18.111687 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.111667    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-kube-api-access-nj6j9" pod "31c4c212-d4ce-4b23-be07-a85fb4ded4f7" (UID: "31c4c212-d4ce-4b23-be07-a85fb4ded4f7"). InnerVolumeSpecName "kube-api-access-nj6j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:18.211675 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.211642    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.211675 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.211668    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.211675 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.211677    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.211675 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.211686    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.211952 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.211695    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj6j9\" (UniqueName: \"kubernetes.io/projected/31c4c212-d4ce-4b23-be07-a85fb4ded4f7-kube-api-access-nj6j9\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.680751 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.680710    4912 generic.go:356] "Generic (PLEG): container finished" podID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerID="0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec" exitCode=0
Nov 02 02:03:18.680935 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.680800    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"
Nov 02 02:03:18.680935 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.680792    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" event={"ID":"31c4c212-d4ce-4b23-be07-a85fb4ded4f7","Type":"ContainerDied","Data":"0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec"}
Nov 02 02:03:18.681011 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.680965    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b" event={"ID":"31c4c212-d4ce-4b23-be07-a85fb4ded4f7","Type":"ContainerDied","Data":"47ebf56561b763e18286ad1570bc2ac7fd1832fb2301cc301a26996be79a8a2c"}
Nov 02 02:03:18.681011 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.680997    4912 scope.go:122] "RemoveContainer" containerID="0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec"
Nov 02 02:03:18.694112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.694045    4912 scope.go:122] "RemoveContainer" containerID="0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec"
Nov 02 02:03:18.694487 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:03:18.694464    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec\": container with ID starting with 0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec not found: ID does not exist" containerID="0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec"
Nov 02 02:03:18.694576 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.694499    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec"} err="failed to get container status \"0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec\": rpc error: code = NotFound desc = could not find container \"0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec\": container with ID starting with 0add1afb231acd830f8be8537dd61548847be109b399674e3f74942a45beffec not found: ID does not exist"
Nov 02 02:03:18.734012 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.733977    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:18.847399 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.847365    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"]
Nov 02 02:03:18.898987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:18.898949    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rss7b"]
Nov 02 02:03:19.228268 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:19.228237    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" path="/var/lib/kubelet/pods/31c4c212-d4ce-4b23-be07-a85fb4ded4f7/volumes"
Nov 02 02:03:21.219790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:21.219760    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:22.354775 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:22.354741    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:22.576945 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:22.576912    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:22.839125 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:22.839094    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:23.600899 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:23.600863    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:23.836697 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:23.836630    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:24.194463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:24.194425    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:24.371971 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:24.371930    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:24.549849 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:24.549754    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:27.124933 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:27.124903    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:27.326586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:27.326551    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:27.507555 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:27.507419    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:27.931153 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:27.927951    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:28.237378 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:28.237294    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:28.420819 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:28.420790    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:29.865074 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:29.865018    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:30.438721 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:30.438685    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:30.664616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:30.664585    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:35.153744 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:35.153706    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:37.695993 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:37.695953    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:37.880885 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:37.880824    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:38.372769 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.372393    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"]
Nov 02 02:03:38.374130 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.374101    4912 cpu_manager.go:414] "containerMap: removing container" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy"
Nov 02 02:03:38.375834 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.375814    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy"
Nov 02 02:03:38.376260 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.376242    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c4c212-d4ce-4b23-be07-a85fb4ded4f7" containerName="istio-proxy"
Nov 02 02:03:38.387293 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.387266    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"]
Nov 02 02:03:38.389890 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.389868    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:38.493064 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.493022    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d1405a\""
Nov 02 02:03:38.493064 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.493040    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d1405a-dockercfg-b2l9v\""
Nov 02 02:03:38.502794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.502764    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdg2c\" (UniqueName: \"kubernetes.io/projected/d24013f9-f96a-4209-a464-18d6d87c22f1-kube-api-access-hdg2c\") pod \"payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv\" (UID: \"d24013f9-f96a-4209-a464-18d6d87c22f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:38.503101 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.503080    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24013f9-f96a-4209-a464-18d6d87c22f1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv\" (UID: \"d24013f9-f96a-4209-a464-18d6d87c22f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:38.604879 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.604844    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24013f9-f96a-4209-a464-18d6d87c22f1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv\" (UID: \"d24013f9-f96a-4209-a464-18d6d87c22f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:38.605107 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.604895    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdg2c\" (UniqueName: \"kubernetes.io/projected/d24013f9-f96a-4209-a464-18d6d87c22f1-kube-api-access-hdg2c\") pod \"payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv\" (UID: \"d24013f9-f96a-4209-a464-18d6d87c22f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:38.605652 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.605632    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24013f9-f96a-4209-a464-18d6d87c22f1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv\" (UID: \"d24013f9-f96a-4209-a464-18d6d87c22f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:38.665727 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.665665    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdg2c\" (UniqueName: \"kubernetes.io/projected/d24013f9-f96a-4209-a464-18d6d87c22f1-kube-api-access-hdg2c\") pod \"payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv\" (UID: \"d24013f9-f96a-4209-a464-18d6d87c22f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:38.715340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:38.715306    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:39.104896 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:03:39.104861    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24013f9_f96a_4209_a464_18d6d87c22f1.slice/crio-02537aa81c8bbb7e6cf4c69f8bf560ba87ba809f57d37014bec81c3f3fd16041 WatchSource:0}: Error finding container 02537aa81c8bbb7e6cf4c69f8bf560ba87ba809f57d37014bec81c3f3fd16041: Status 404 returned error can't find the container with id 02537aa81c8bbb7e6cf4c69f8bf560ba87ba809f57d37014bec81c3f3fd16041
Nov 02 02:03:39.105407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:39.105301    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"]
Nov 02 02:03:39.910132 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:39.910092    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv" event={"ID":"d24013f9-f96a-4209-a464-18d6d87c22f1","Type":"ContainerStarted","Data":"de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e"}
Nov 02 02:03:39.910574 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:39.910157    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv" event={"ID":"d24013f9-f96a-4209-a464-18d6d87c22f1","Type":"ContainerStarted","Data":"02537aa81c8bbb7e6cf4c69f8bf560ba87ba809f57d37014bec81c3f3fd16041"}
Nov 02 02:03:39.910574 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:39.910184    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:40.027190 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:40.027136    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv" podStartSLOduration=2.027122566 podStartE2EDuration="2.027122566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:40.024533035 +0000 UTC m=+1912.929998393" watchObservedRunningTime="2026-11-02 02:03:40.027122566 +0000 UTC m=+1912.932587921"
Nov 02 02:03:40.037525 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:40.037494    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:42.139142 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:03:42.139099    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:03:44.969143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:44.969104    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:48.744134 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:48.734666    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:48.945277 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:48.945233    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:50.084953 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:50.084901    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:50.410748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:50.410718    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:50.764848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:50.764682    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:50.922811 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:50.922760    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:03:51.229221 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:51.229177    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:55.298586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:55.298550    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:55.604706 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:55.604620    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:03:55.792768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:03:55.792731    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:00.652274 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:00.652170    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:00.882948 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:00.882783    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:01.071137 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.070946    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:01.420453 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.420419    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"]
Nov 02 02:04:01.436454 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.436425    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:01.475312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.475274    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"]
Nov 02 02:04:01.515907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.515877    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-aa1597\""
Nov 02 02:04:01.516133 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.516042    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-aa1597-dockercfg-fjdrk\""
Nov 02 02:04:01.524833 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.524799    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd02b17-4d56-4d3c-b93b-71278a907bca-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79\" (UID: \"2cd02b17-4d56-4d3c-b93b-71278a907bca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:01.524996 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.524837    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2flnt\" (UniqueName: \"kubernetes.io/projected/2cd02b17-4d56-4d3c-b93b-71278a907bca-kube-api-access-2flnt\") pod \"payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79\" (UID: \"2cd02b17-4d56-4d3c-b93b-71278a907bca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:01.626333 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.626297    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd02b17-4d56-4d3c-b93b-71278a907bca-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79\" (UID: \"2cd02b17-4d56-4d3c-b93b-71278a907bca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:01.626333 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.626335    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2flnt\" (UniqueName: \"kubernetes.io/projected/2cd02b17-4d56-4d3c-b93b-71278a907bca-kube-api-access-2flnt\") pod \"payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79\" (UID: \"2cd02b17-4d56-4d3c-b93b-71278a907bca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:01.627170 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.627145    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd02b17-4d56-4d3c-b93b-71278a907bca-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79\" (UID: \"2cd02b17-4d56-4d3c-b93b-71278a907bca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:01.691313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.691233    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2flnt\" (UniqueName: \"kubernetes.io/projected/2cd02b17-4d56-4d3c-b93b-71278a907bca-kube-api-access-2flnt\") pod \"payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79\" (UID: \"2cd02b17-4d56-4d3c-b93b-71278a907bca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:01.782693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:01.782662    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:02.183891 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:02.183824    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"]
Nov 02 02:04:02.185730 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:04:02.185694    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd02b17_4d56_4d3c_b93b_71278a907bca.slice/crio-d88105ad05291d59cd3e1fba8e2c00c0441b63e822e4317ad94a9b2c9e921083 WatchSource:0}: Error finding container d88105ad05291d59cd3e1fba8e2c00c0441b63e822e4317ad94a9b2c9e921083: Status 404 returned error can't find the container with id d88105ad05291d59cd3e1fba8e2c00c0441b63e822e4317ad94a9b2c9e921083
Nov 02 02:04:03.155288 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:03.155241    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79" event={"ID":"2cd02b17-4d56-4d3c-b93b-71278a907bca","Type":"ContainerStarted","Data":"502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba"}
Nov 02 02:04:03.155734 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:03.155354    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:03.155734 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:03.155379    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79" event={"ID":"2cd02b17-4d56-4d3c-b93b-71278a907bca","Type":"ContainerStarted","Data":"d88105ad05291d59cd3e1fba8e2c00c0441b63e822e4317ad94a9b2c9e921083"}
Nov 02 02:04:03.272578 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:03.272521    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79" podStartSLOduration=2.272506192 podStartE2EDuration="2.272506192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:03.270620626 +0000 UTC m=+1936.176085985" watchObservedRunningTime="2026-11-02 02:04:03.272506192 +0000 UTC m=+1936.177971549"
Nov 02 02:04:05.548491 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:05.548458    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:06.114823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:06.114784    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"]
Nov 02 02:04:06.115065 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:06.115030    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy" containerID="cri-o://3f7af1341c149adf804d0a8c8ec66f3f1c7d1b712db3d473f9ecf2c89202a0c6" gracePeriod=30
Nov 02 02:04:10.593871 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:10.593827    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:10.818781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:10.818745    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:11.018757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.018720    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:11.234990 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.234964    4912 generic.go:356] "Generic (PLEG): container finished" podID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerID="3f7af1341c149adf804d0a8c8ec66f3f1c7d1b712db3d473f9ecf2c89202a0c6" exitCode=0
Nov 02 02:04:11.235160 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.235041    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" event={"ID":"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c","Type":"ContainerDied","Data":"3f7af1341c149adf804d0a8c8ec66f3f1c7d1b712db3d473f9ecf2c89202a0c6"}
Nov 02 02:04:11.427643 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.427617    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 02:04:11.514699 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.514670    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-token\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-token\") pod \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") "
Nov 02 02:04:11.514885 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.514744    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istiod-ca-cert\") pod \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") "
Nov 02 02:04:11.514885 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.514859    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-credential-socket\") pod \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") "
Nov 02 02:04:11.515013 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.514959    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-socket\") pod \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") "
Nov 02 02:04:11.515144 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515128    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-certs\") pod \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") "
Nov 02 02:04:11.515219 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515165    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-podinfo\") pod \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") "
Nov 02 02:04:11.515219 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515125    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-credential-socket" pod "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" (UID: "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:11.515323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515214    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-socket" pod "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" (UID: "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:11.515323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515227    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-data\") pod \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") "
Nov 02 02:04:11.515323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515154    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istiod-ca-cert" pod "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" (UID: "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:11.515479 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515322    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-envoy\") pod \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") "
Nov 02 02:04:11.515479 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515347    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-certs" pod "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" (UID: "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:11.515479 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515360    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-kube-api-access-674px\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-kube-api-access-674px\") pod \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\" (UID: \"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c\") "
Nov 02 02:04:11.515479 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.515423    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-data" pod "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" (UID: "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:11.516105 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.516080    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:11.516234 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.516109    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:11.516234 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.516125    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:11.516234 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.516138    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:11.516423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.516222    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:11.517302 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.517277    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-token" pod "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" (UID: "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:11.517535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.517514    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-podinfo" pod "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" (UID: "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:04:11.517595 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.517529    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-envoy" pod "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" (UID: "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:11.517699 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.517685    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-kube-api-access-674px" pod "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" (UID: "5d487f28-5c3b-4f6a-8259-b5f9c36fc88c"). InnerVolumeSpecName "kube-api-access-674px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:11.618103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.618074    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:11.618499 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.618133    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:11.618499 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.618150    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:11.618499 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:11.618165    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-674px\" (UniqueName: \"kubernetes.io/projected/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c-kube-api-access-674px\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:12.140845 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:04:12.140806    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:04:12.245838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:12.245807    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"
Nov 02 02:04:12.246047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:12.245835    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2" event={"ID":"5d487f28-5c3b-4f6a-8259-b5f9c36fc88c","Type":"ContainerDied","Data":"56b76f4496b1dc47b0cc0a7ceafb284a5a865b8b11f1d841c505f0f6e2e3d563"}
Nov 02 02:04:12.246047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:12.245907    4912 scope.go:122] "RemoveContainer" containerID="3f7af1341c149adf804d0a8c8ec66f3f1c7d1b712db3d473f9ecf2c89202a0c6"
Nov 02 02:04:12.462836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:12.462745    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"]
Nov 02 02:04:12.512682 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:12.512648    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-pglx2"]
Nov 02 02:04:13.228046 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:13.228013    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" path="/var/lib/kubelet/pods/5d487f28-5c3b-4f6a-8259-b5f9c36fc88c/volumes"
Nov 02 02:04:14.169232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:14.169204    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:04:14.342493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:14.342452    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"]
Nov 02 02:04:14.342937 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:14.342716    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy" containerID="cri-o://a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0" gracePeriod=30
Nov 02 02:04:15.675716 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:15.675680    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:16.229442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:16.229405    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:04:16.229626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:16.229463    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:04:19.644959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.644930    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 02:04:19.751878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.751844    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-envoy\") pod \"73c66bf0-994e-49a2-b485-d3dcff93447e\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") "
Nov 02 02:04:19.752103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.751897    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-kube-api-access-dx69b\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-kube-api-access-dx69b\") pod \"73c66bf0-994e-49a2-b485-d3dcff93447e\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") "
Nov 02 02:04:19.752103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.751956    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-data\") pod \"73c66bf0-994e-49a2-b485-d3dcff93447e\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") "
Nov 02 02:04:19.752103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752004    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-credential-socket\") pod \"73c66bf0-994e-49a2-b485-d3dcff93447e\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") "
Nov 02 02:04:19.752103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752048    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-socket\") pod \"73c66bf0-994e-49a2-b485-d3dcff93447e\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") "
Nov 02 02:04:19.752343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752156    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-certs\") pod \"73c66bf0-994e-49a2-b485-d3dcff93447e\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") "
Nov 02 02:04:19.752343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752207    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-podinfo\") pod \"73c66bf0-994e-49a2-b485-d3dcff93447e\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") "
Nov 02 02:04:19.752343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752255    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-token\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-token\") pod \"73c66bf0-994e-49a2-b485-d3dcff93447e\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") "
Nov 02 02:04:19.752343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752283    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c66bf0-994e-49a2-b485-d3dcff93447e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c66bf0-994e-49a2-b485-d3dcff93447e-istiod-ca-cert\") pod \"73c66bf0-994e-49a2-b485-d3dcff93447e\" (UID: \"73c66bf0-994e-49a2-b485-d3dcff93447e\") "
Nov 02 02:04:19.752548 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752353    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-credential-socket" pod "73c66bf0-994e-49a2-b485-d3dcff93447e" (UID: "73c66bf0-994e-49a2-b485-d3dcff93447e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:19.752548 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752456    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-certs" pod "73c66bf0-994e-49a2-b485-d3dcff93447e" (UID: "73c66bf0-994e-49a2-b485-d3dcff93447e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:19.752654 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752616    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-socket" pod "73c66bf0-994e-49a2-b485-d3dcff93447e" (UID: "73c66bf0-994e-49a2-b485-d3dcff93447e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:19.752940 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752915    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c66bf0-994e-49a2-b485-d3dcff93447e-istiod-ca-cert" pod "73c66bf0-994e-49a2-b485-d3dcff93447e" (UID: "73c66bf0-994e-49a2-b485-d3dcff93447e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:19.753106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.752980    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-data" pod "73c66bf0-994e-49a2-b485-d3dcff93447e" (UID: "73c66bf0-994e-49a2-b485-d3dcff93447e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:19.753194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.753127    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:19.753194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.753140    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:19.753194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.753149    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:19.753194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.753161    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:19.753194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.753179    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c66bf0-994e-49a2-b485-d3dcff93447e-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:19.754658 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.754618    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-envoy" pod "73c66bf0-994e-49a2-b485-d3dcff93447e" (UID: "73c66bf0-994e-49a2-b485-d3dcff93447e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:19.754829 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.754768    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-kube-api-access-dx69b" pod "73c66bf0-994e-49a2-b485-d3dcff93447e" (UID: "73c66bf0-994e-49a2-b485-d3dcff93447e"). InnerVolumeSpecName "kube-api-access-dx69b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:19.755226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.755203    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-token" pod "73c66bf0-994e-49a2-b485-d3dcff93447e" (UID: "73c66bf0-994e-49a2-b485-d3dcff93447e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:19.755407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.755382    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-podinfo" pod "73c66bf0-994e-49a2-b485-d3dcff93447e" (UID: "73c66bf0-994e-49a2-b485-d3dcff93447e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:04:19.855052 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.855017    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:19.855052 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.855102    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:19.855052 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.855122    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c66bf0-994e-49a2-b485-d3dcff93447e-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:19.855403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:19.855138    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx69b\" (UniqueName: \"kubernetes.io/projected/73c66bf0-994e-49a2-b485-d3dcff93447e-kube-api-access-dx69b\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:20.324613 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.324574    4912 generic.go:356] "Generic (PLEG): container finished" podID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerID="a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0" exitCode=0
Nov 02 02:04:20.324891 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.324622    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"
Nov 02 02:04:20.324891 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.324655    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" event={"ID":"73c66bf0-994e-49a2-b485-d3dcff93447e","Type":"ContainerDied","Data":"a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0"}
Nov 02 02:04:20.324891 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.324713    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk" event={"ID":"73c66bf0-994e-49a2-b485-d3dcff93447e","Type":"ContainerDied","Data":"80282909059eebafa6819bd1832a2f91137f86a8d283df53697264c8ed006820"}
Nov 02 02:04:20.324891 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.324742    4912 scope.go:122] "RemoveContainer" containerID="a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0"
Nov 02 02:04:20.348851 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.348830    4912 scope.go:122] "RemoveContainer" containerID="a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0"
Nov 02 02:04:20.349200 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:04:20.349174    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0\": container with ID starting with a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0 not found: ID does not exist" containerID="a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0"
Nov 02 02:04:20.349309 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.349205    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0"} err="failed to get container status \"a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0\": rpc error: code = NotFound desc = could not find container \"a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0\": container with ID starting with a57993f9400cd4ff5d056ba07a862b7256b72979c0836f786087dd65caa7b1c0 not found: ID does not exist"
Nov 02 02:04:20.485244 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.485203    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:20.543376 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.543340    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"]
Nov 02 02:04:20.590387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:20.590312    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-nzlmk"]
Nov 02 02:04:21.228215 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:21.228183    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" path="/var/lib/kubelet/pods/73c66bf0-994e-49a2-b485-d3dcff93447e/volumes"
Nov 02 02:04:25.446682 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:25.446643    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:30.089104 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:30.089032    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:34.807919 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:34.807878    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:39.287175 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.287137    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"]
Nov 02 02:04:39.288335 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.288313    4912 cpu_manager.go:414] "containerMap: removing container" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy"
Nov 02 02:04:39.288335 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.288335    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy"
Nov 02 02:04:39.288511 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.288352    4912 cpu_manager.go:414] "containerMap: removing container" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy"
Nov 02 02:04:39.288511 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.288361    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy"
Nov 02 02:04:39.288614 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.288526    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d487f28-5c3b-4f6a-8259-b5f9c36fc88c" containerName="istio-proxy"
Nov 02 02:04:39.288614 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.288538    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c66bf0-994e-49a2-b485-d3dcff93447e" containerName="istio-proxy"
Nov 02 02:04:39.292434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.292413    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.347366 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.347331    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"]
Nov 02 02:04:39.376945 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.376911    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-47eb54-serving-cert\""
Nov 02 02:04:39.456997 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.456963    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ad0c3-b787-40e4-a50f-7442d9af30c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.457191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.457135    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.457237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.457201    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.457354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.457339    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwgtz\" (UniqueName: \"kubernetes.io/projected/e27ad0c3-b787-40e4-a50f-7442d9af30c9-kube-api-access-hwgtz\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.503648 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.503613    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:39.560209 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.560110    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ad0c3-b787-40e4-a50f-7442d9af30c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.560398 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.560249    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.560398 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.560329    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.573459 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.564991    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ad0c3-b787-40e4-a50f-7442d9af30c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.573909 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:04:39.565383    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-serving-cert: secret "maas-api-e2e-shared-b-w0-47eb54-serving-cert" not found
Nov 02 02:04:39.577835 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.577769    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwgtz\" (UniqueName: \"kubernetes.io/projected/e27ad0c3-b787-40e4-a50f-7442d9af30c9-kube-api-access-hwgtz\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.578593 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:04:39.578095    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls podName:e27ad0c3-b787-40e4-a50f-7442d9af30c9 nodeName:}" failed. No retries permitted until 2026-11-02 02:04:40.078050629 +0000 UTC m=+1972.983515978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls") pod "maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj" (UID: "e27ad0c3-b787-40e4-a50f-7442d9af30c9") : secret "maas-api-e2e-shared-b-w0-47eb54-serving-cert" not found
Nov 02 02:04:39.584483 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.584458    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.638983 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.638953    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwgtz\" (UniqueName: \"kubernetes.io/projected/e27ad0c3-b787-40e4-a50f-7442d9af30c9-kube-api-access-hwgtz\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:39.687478 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.687433    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:39.867287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:39.867251    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:40.088960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:40.088923    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:40.091827 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:40.091800    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:40.207853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:40.207824    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:40.598160 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:40.598132    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"]
Nov 02 02:04:40.599095 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:04:40.599025    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27ad0c3_b787_40e4_a50f_7442d9af30c9.slice/crio-280fd30f541d30f379c3d5f49e370f1a66074c51c3f47bbc06150281317886d5 WatchSource:0}: Error finding container 280fd30f541d30f379c3d5f49e370f1a66074c51c3f47bbc06150281317886d5: Status 404 returned error can't find the container with id 280fd30f541d30f379c3d5f49e370f1a66074c51c3f47bbc06150281317886d5
Nov 02 02:04:41.513502 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:41.513471    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj" event={"ID":"e27ad0c3-b787-40e4-a50f-7442d9af30c9","Type":"ContainerStarted","Data":"280fd30f541d30f379c3d5f49e370f1a66074c51c3f47bbc06150281317886d5"}
Nov 02 02:04:42.148542 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:04:42.148502    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:04:42.525360 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:42.525278    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj" event={"ID":"e27ad0c3-b787-40e4-a50f-7442d9af30c9","Type":"ContainerStarted","Data":"57aaa11a66a98768db38a9edc99026fa7a8254747273c56ec7b5281cad0ac650"}
Nov 02 02:04:42.527617 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:42.525357    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:42.646919 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:42.646869    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj" podStartSLOduration=2.818007853 podStartE2EDuration="3.646855276s" totalImagesPullingTime="828.847423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:42.642249603 +0000 UTC m=+1975.547714962" watchObservedRunningTime="2026-11-02 02:04:42.646855276 +0000 UTC m=+1975.552320632"
Nov 02 02:04:44.032860 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:44.032825    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:44.451151 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:44.451113    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:48.540715 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:48.540687    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:04:49.095097 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:49.095047    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:49.278570 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:49.278534    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:53.824584 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:53.824551    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:54.019968 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:54.019916    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:58.813588 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:58.813549    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:59.002617 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:59.001596    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:04:59.178201 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:04:59.178158    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:02.653428 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.653385    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"]
Nov 02 02:05:02.659388 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.659362    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.714735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.714703    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"]
Nov 02 02:05:02.757767 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.757721    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:05:02.757960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.757750    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:05:02.765705 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.765677    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd3971-555fz\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.765855 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.765787    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd3971-555fz\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.765951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.765932    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxqk\" (UniqueName: \"kubernetes.io/projected/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-kube-api-access-9dxqk\") pod \"maas-api-revoke-keys-e2e-api-a-bd3971-555fz\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.867744 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.867707    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxqk\" (UniqueName: \"kubernetes.io/projected/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-kube-api-access-9dxqk\") pod \"maas-api-revoke-keys-e2e-api-a-bd3971-555fz\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.867965 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.867945    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd3971-555fz\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.868121 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.868103    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd3971-555fz\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.868983 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.868854    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd3971-555fz\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.870828 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.870793    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd3971-555fz\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.928492 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.928419    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxqk\" (UniqueName: \"kubernetes.io/projected/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-kube-api-access-9dxqk\") pod \"maas-api-revoke-keys-e2e-api-a-bd3971-555fz\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:02.975716 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:02.975681    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:03.376100 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:03.376035    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"]
Nov 02 02:05:03.727166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:03.727084    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz" event={"ID":"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb","Type":"ContainerStarted","Data":"4c64d8bc3b62d2e7da16f03e3299f82005496d04e89545d3ee3160fb14aaa1f7"}
Nov 02 02:05:04.052705 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:04.052625    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:04.228102 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:04.228049    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:04.405601 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:04.405551    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:04.737462 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:04.737431    4912 generic.go:356] "Generic (PLEG): container finished" podID="82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb" containerID="43a727922f5c5c6ddb83b15c32d8d65f200f405b01e670f4a37bc1df88fd5360" exitCode=0
Nov 02 02:05:04.737862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:04.737518    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz" event={"ID":"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb","Type":"ContainerDied","Data":"43a727922f5c5c6ddb83b15c32d8d65f200f405b01e670f4a37bc1df88fd5360"}
Nov 02 02:05:05.139775 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.139745    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"]
Nov 02 02:05:05.144415 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.144398    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.204398 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.204259    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"]
Nov 02 02:05:05.326914 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.326878    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.327165 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.326993    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.327165 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.327150    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xjb\" (UniqueName: \"kubernetes.io/projected/dcd2e80c-04ca-425f-925b-e56edde1023c-kube-api-access-d4xjb\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.428435 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.428352    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.428590 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.428459    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.428633 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.428619    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xjb\" (UniqueName: \"kubernetes.io/projected/dcd2e80c-04ca-425f-925b-e56edde1023c-kube-api-access-d4xjb\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.429259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.429170    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.430917 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.430893    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.489585 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.489545    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xjb\" (UniqueName: \"kubernetes.io/projected/dcd2e80c-04ca-425f-925b-e56edde1023c-kube-api-access-d4xjb\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.760876 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.760841    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:05.987709 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:05.987677    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:06.149093 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.147721    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-openshift-ca\") pod \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") "
Nov 02 02:05:06.149093 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.147863    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-xks-ca\") pod \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") "
Nov 02 02:05:06.149093 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.148000    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-kube-api-access-9dxqk\" (UniqueName: \"kubernetes.io/projected/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-kube-api-access-9dxqk\") pod \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\" (UID: \"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb\") "
Nov 02 02:05:06.156204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.150187    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-openshift-ca" pod "82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb" (UID: "82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:06.156204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.152913    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-xks-ca" pod "82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb" (UID: "82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:05:06.156204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.153193    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:06.156204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.153215    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:06.178090 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.174501    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-kube-api-access-9dxqk" pod "82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb" (UID: "82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb"). InnerVolumeSpecName "kube-api-access-9dxqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:06.220239 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.220208    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"]
Nov 02 02:05:06.220579 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:05:06.220552    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd2e80c_04ca_425f_925b_e56edde1023c.slice/crio-1d6e645e6b0def7834d594393abf1c0049ffa0872c3a479fa276a48a70888372 WatchSource:0}: Error finding container 1d6e645e6b0def7834d594393abf1c0049ffa0872c3a479fa276a48a70888372: Status 404 returned error can't find the container with id 1d6e645e6b0def7834d594393abf1c0049ffa0872c3a479fa276a48a70888372
Nov 02 02:05:06.254260 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.254231    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxqk\" (UniqueName: \"kubernetes.io/projected/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb-kube-api-access-9dxqk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:06.770204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.770174    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv" event={"ID":"dcd2e80c-04ca-425f-925b-e56edde1023c","Type":"ContainerStarted","Data":"9cec03395b9baad35a753223b71bdf7641b2f6e9b452bed8c8366de817b69c5e"}
Nov 02 02:05:06.770982 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.770235    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv" event={"ID":"dcd2e80c-04ca-425f-925b-e56edde1023c","Type":"ContainerStarted","Data":"1d6e645e6b0def7834d594393abf1c0049ffa0872c3a479fa276a48a70888372"}
Nov 02 02:05:06.771874 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.771849    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"
Nov 02 02:05:06.771874 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.771867    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz" event={"ID":"82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb","Type":"ContainerDied","Data":"4c64d8bc3b62d2e7da16f03e3299f82005496d04e89545d3ee3160fb14aaa1f7"}
Nov 02 02:05:06.772095 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.771901    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c64d8bc3b62d2e7da16f03e3299f82005496d04e89545d3ee3160fb14aaa1f7"
Nov 02 02:05:06.893139 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:06.893083    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv" podStartSLOduration=1.893050053 podStartE2EDuration="1.893050053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:05:06.885253688 +0000 UTC m=+1999.790719047" watchObservedRunningTime="2026-11-02 02:05:06.893050053 +0000 UTC m=+1999.798515412"
Nov 02 02:05:07.783112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:07.782984    4912 generic.go:356] "Generic (PLEG): container finished" podID="dcd2e80c-04ca-425f-925b-e56edde1023c" containerID="9cec03395b9baad35a753223b71bdf7641b2f6e9b452bed8c8366de817b69c5e" exitCode=0
Nov 02 02:05:07.783514 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:07.783103    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv" event={"ID":"dcd2e80c-04ca-425f-925b-e56edde1023c","Type":"ContainerDied","Data":"9cec03395b9baad35a753223b71bdf7641b2f6e9b452bed8c8366de817b69c5e"}
Nov 02 02:05:08.979066 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:08.979019    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:09.103382 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.102825    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-xks-ca\") pod \"dcd2e80c-04ca-425f-925b-e56edde1023c\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") "
Nov 02 02:05:09.103382 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.102929    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-openshift-ca\") pod \"dcd2e80c-04ca-425f-925b-e56edde1023c\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") "
Nov 02 02:05:09.103382 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.103143    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd2e80c-04ca-425f-925b-e56edde1023c-kube-api-access-d4xjb\" (UniqueName: \"kubernetes.io/projected/dcd2e80c-04ca-425f-925b-e56edde1023c-kube-api-access-d4xjb\") pod \"dcd2e80c-04ca-425f-925b-e56edde1023c\" (UID: \"dcd2e80c-04ca-425f-925b-e56edde1023c\") "
Nov 02 02:05:09.103382 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.103257    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-openshift-ca" pod "dcd2e80c-04ca-425f-925b-e56edde1023c" (UID: "dcd2e80c-04ca-425f-925b-e56edde1023c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:09.104092 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.103886    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:09.106449 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.106412    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-xks-ca" pod "dcd2e80c-04ca-425f-925b-e56edde1023c" (UID: "dcd2e80c-04ca-425f-925b-e56edde1023c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:05:09.106803 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.106781    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd2e80c-04ca-425f-925b-e56edde1023c-kube-api-access-d4xjb" pod "dcd2e80c-04ca-425f-925b-e56edde1023c" (UID: "dcd2e80c-04ca-425f-925b-e56edde1023c"). InnerVolumeSpecName "kube-api-access-d4xjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:09.205907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.205871    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd2e80c-04ca-425f-925b-e56edde1023c-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:09.205907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.205901    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xjb\" (UniqueName: \"kubernetes.io/projected/dcd2e80c-04ca-425f-925b-e56edde1023c-kube-api-access-d4xjb\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:09.378457 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.378422    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:09.811136 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.811008    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"
Nov 02 02:05:09.811136 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.811040    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv" event={"ID":"dcd2e80c-04ca-425f-925b-e56edde1023c","Type":"ContainerDied","Data":"1d6e645e6b0def7834d594393abf1c0049ffa0872c3a479fa276a48a70888372"}
Nov 02 02:05:09.811367 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:09.811157    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d6e645e6b0def7834d594393abf1c0049ffa0872c3a479fa276a48a70888372"
Nov 02 02:05:12.138177 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:05:12.138141    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:05:14.377426 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:14.377379    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:15.404771 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:15.404633    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:15.587508 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:15.587457    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:17.465142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:17.465051    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"]
Nov 02 02:05:17.577895 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:17.577813    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"]
Nov 02 02:05:17.871237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:17.871200    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:18.060717 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:18.060681    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:19.522415 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:19.522372    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:22.465456 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:22.465397    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb" podUID="592eef38-f316-4f9c-9a5d-01db442dcf74" containerName="payload-processing" containerID="cri-o://495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7" gracePeriod=25
Nov 02 02:05:22.578526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:22.578439    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z" podUID="a6baaf7a-c642-4575-84a4-7ad1035bcce6" containerName="payload-pre-processing" containerID="cri-o://7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb" gracePeriod=25
Nov 02 02:05:24.233278 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:24.233240    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:28.642809 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:28.642734    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"]
Nov 02 02:05:28.643233 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:28.642949    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy" containerID="cri-o://2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9" gracePeriod=30
Nov 02 02:05:29.151111 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:29.150566    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:30.887490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:30.887438    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb" podUID="592eef38-f316-4f9c-9a5d-01db442dcf74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:05:30.887969 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:30.887488    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z" podUID="a6baaf7a-c642-4575-84a4-7ad1035bcce6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:05:31.287981 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:31.287850    4912 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:05:31.288176 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:31.287991    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:05:33.205307 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.205273    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:33.388592 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.388555    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:33.668350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.668317    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"]
Nov 02 02:05:33.669493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.669472    4912 cpu_manager.go:414] "containerMap: removing container" podUID="82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb" containerName="revoke-keys"
Nov 02 02:05:33.669572 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.669497    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb" containerName="revoke-keys"
Nov 02 02:05:33.669572 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.669526    4912 cpu_manager.go:414] "containerMap: removing container" podUID="dcd2e80c-04ca-425f-925b-e56edde1023c" containerName="revoke-keys"
Nov 02 02:05:33.669572 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.669536    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd2e80c-04ca-425f-925b-e56edde1023c" containerName="revoke-keys"
Nov 02 02:05:33.669758 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.669746    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb" containerName="revoke-keys"
Nov 02 02:05:33.669794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.669764    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd2e80c-04ca-425f-925b-e56edde1023c" containerName="revoke-keys"
Nov 02 02:05:33.674375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.674348    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.729194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.729165    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"]
Nov 02 02:05:33.738411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.738379    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d15873-c164-4c41-aac9-bff2fa0f72fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.738556 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.738440    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-metrics-certs\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.738556 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.738533    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.738697 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.738623    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpl9\" (UniqueName: \"kubernetes.io/projected/b7d15873-c164-4c41-aac9-bff2fa0f72fe-kube-api-access-ccpl9\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.771836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.771263    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-aa1597-serving-cert\""
Nov 02 02:05:33.840518 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.840479    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.840699 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.840630    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpl9\" (UniqueName: \"kubernetes.io/projected/b7d15873-c164-4c41-aac9-bff2fa0f72fe-kube-api-access-ccpl9\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.840699 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:05:33.840651    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-serving-cert: secret "maas-api-e2e-inf-a-aa1597-serving-cert" not found
Nov 02 02:05:33.840796 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:05:33.840742    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls podName:b7d15873-c164-4c41-aac9-bff2fa0f72fe nodeName:}" failed. No retries permitted until 2026-11-02 02:05:34.340719266 +0000 UTC m=+2027.246184601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls") pod "maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr" (UID: "b7d15873-c164-4c41-aac9-bff2fa0f72fe") : secret "maas-api-e2e-inf-a-aa1597-serving-cert" not found
Nov 02 02:05:33.840960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.840936    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d15873-c164-4c41-aac9-bff2fa0f72fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.841034 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.841003    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-metrics-certs\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.841732 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.841698    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d15873-c164-4c41-aac9-bff2fa0f72fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.843671 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.843653    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-metrics-certs\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:33.879284 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.879246    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:33.930211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:33.930100    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpl9\" (UniqueName: \"kubernetes.io/projected/b7d15873-c164-4c41-aac9-bff2fa0f72fe-kube-api-access-ccpl9\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:34.023510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.023486    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 02:05:34.038199 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.038171    4912 generic.go:356] "Generic (PLEG): container finished" podID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerID="2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9" exitCode=0
Nov 02 02:05:34.038350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.038213    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"
Nov 02 02:05:34.038350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.038251    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" event={"ID":"94764ea2-8bb7-4ce7-897a-fd0a1886f97f","Type":"ContainerDied","Data":"2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9"}
Nov 02 02:05:34.038350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.038311    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v" event={"ID":"94764ea2-8bb7-4ce7-897a-fd0a1886f97f","Type":"ContainerDied","Data":"a111fc89376ff5e280b21605af3ddf2cc4330bc35aebba078d505c4f4f82458e"}
Nov 02 02:05:34.038350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.038341    4912 scope.go:122] "RemoveContainer" containerID="2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9"
Nov 02 02:05:34.051407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.051377    4912 scope.go:122] "RemoveContainer" containerID="2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9"
Nov 02 02:05:34.051726 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:05:34.051705    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9\": container with ID starting with 2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9 not found: ID does not exist" containerID="2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9"
Nov 02 02:05:34.051846 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.051736    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9"} err="failed to get container status \"2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9\": rpc error: code = NotFound desc = could not find container \"2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9\": container with ID starting with 2afcdb841a03197e68045ac39ae9454530c33e5c5aaaa7d3eeac1e94f0acb8b9 not found: ID does not exist"
Nov 02 02:05:34.077257 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.077227    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.162725    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-token\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-token\") pod \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") "
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.162833    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-envoy\") pod \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") "
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.162909    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-certs\") pod \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") "
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.163035    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-podinfo\") pod \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") "
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.163109    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-data\") pod \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") "
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.163168    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-credential-socket\") pod \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") "
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.163280    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-socket\") pod \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") "
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.163327    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istiod-ca-cert\") pod \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") "
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.163381    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-kube-api-access-cq5gz\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-kube-api-access-cq5gz\") pod \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\" (UID: \"94764ea2-8bb7-4ce7-897a-fd0a1886f97f\") "
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.163959    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-data" pod "94764ea2-8bb7-4ce7-897a-fd0a1886f97f" (UID: "94764ea2-8bb7-4ce7-897a-fd0a1886f97f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.164160    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.164401    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-certs" pod "94764ea2-8bb7-4ce7-897a-fd0a1886f97f" (UID: "94764ea2-8bb7-4ce7-897a-fd0a1886f97f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.165655    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-socket" pod "94764ea2-8bb7-4ce7-897a-fd0a1886f97f" (UID: "94764ea2-8bb7-4ce7-897a-fd0a1886f97f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.166430    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-credential-socket" pod "94764ea2-8bb7-4ce7-897a-fd0a1886f97f" (UID: "94764ea2-8bb7-4ce7-897a-fd0a1886f97f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.166590    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istiod-ca-cert" pod "94764ea2-8bb7-4ce7-897a-fd0a1886f97f" (UID: "94764ea2-8bb7-4ce7-897a-fd0a1886f97f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:34.168374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.167487    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-podinfo" pod "94764ea2-8bb7-4ce7-897a-fd0a1886f97f" (UID: "94764ea2-8bb7-4ce7-897a-fd0a1886f97f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:05:34.169452 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.168454    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-kube-api-access-cq5gz" pod "94764ea2-8bb7-4ce7-897a-fd0a1886f97f" (UID: "94764ea2-8bb7-4ce7-897a-fd0a1886f97f"). InnerVolumeSpecName "kube-api-access-cq5gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:34.179547 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.179511    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-envoy" pod "94764ea2-8bb7-4ce7-897a-fd0a1886f97f" (UID: "94764ea2-8bb7-4ce7-897a-fd0a1886f97f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:34.182198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.182131    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-token" pod "94764ea2-8bb7-4ce7-897a-fd0a1886f97f" (UID: "94764ea2-8bb7-4ce7-897a-fd0a1886f97f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:34.268584 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.268546    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:34.268584 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.268582    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:34.269279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.268597    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:34.269279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.268612    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:34.269279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.268628    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:34.269279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.268640    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:34.269279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.268654    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:34.269279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.268665    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5gz\" (UniqueName: \"kubernetes.io/projected/94764ea2-8bb7-4ce7-897a-fd0a1886f97f-kube-api-access-cq5gz\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:34.271627 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.271597    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:34.370033 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.370002    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:34.372823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.372796    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:34.461570 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.461484    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"]
Nov 02 02:05:34.468778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.468744    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-6rr5v"]
Nov 02 02:05:34.644259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:34.644225    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:35.035255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:35.035225    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"]
Nov 02 02:05:35.039610 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:35.039586    4912 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:05:35.049429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:35.049401    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr" event={"ID":"b7d15873-c164-4c41-aac9-bff2fa0f72fe","Type":"ContainerStarted","Data":"c215a5900ee3b7ad75a743893ad31d342ca1482d4235f52e9da4fc922fc8d3f3"}
Nov 02 02:05:35.228575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:35.228542    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" path="/var/lib/kubelet/pods/94764ea2-8bb7-4ce7-897a-fd0a1886f97f/volumes"
Nov 02 02:05:35.758503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:35.756433    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:35.933495 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:35.933461    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:37.073521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:37.073489    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr" event={"ID":"b7d15873-c164-4c41-aac9-bff2fa0f72fe","Type":"ContainerStarted","Data":"0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39"}
Nov 02 02:05:37.073962 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:37.073596    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:37.193010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:37.192957    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr" podStartSLOduration=3.229378367 podStartE2EDuration="4.192943352s" totalImagesPullingTime="963.564985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:05:37.190301737 +0000 UTC m=+2030.095767095" watchObservedRunningTime="2026-11-02 02:05:37.192943352 +0000 UTC m=+2030.098408709"
Nov 02 02:05:38.811141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:38.811102    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:40.887684 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:40.887647    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z" podUID="a6baaf7a-c642-4575-84a4-7ad1035bcce6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:05:40.888195 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:40.887699    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb" podUID="592eef38-f316-4f9c-9a5d-01db442dcf74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:05:42.166486 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:05:42.166449    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:05:42.769001 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:42.768972    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:05:42.876073 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:42.876037    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:05:42.917848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:42.917820    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592eef38-f316-4f9c-9a5d-01db442dcf74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592eef38-f316-4f9c-9a5d-01db442dcf74-plugins-config-volume\") pod \"592eef38-f316-4f9c-9a5d-01db442dcf74\" (UID: \"592eef38-f316-4f9c-9a5d-01db442dcf74\") "
Nov 02 02:05:42.918034 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:42.918013    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592eef38-f316-4f9c-9a5d-01db442dcf74-kube-api-access-r89f2\" (UniqueName: \"kubernetes.io/projected/592eef38-f316-4f9c-9a5d-01db442dcf74-kube-api-access-r89f2\") pod \"592eef38-f316-4f9c-9a5d-01db442dcf74\" (UID: \"592eef38-f316-4f9c-9a5d-01db442dcf74\") "
Nov 02 02:05:42.918343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:42.918312    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592eef38-f316-4f9c-9a5d-01db442dcf74-plugins-config-volume" pod "592eef38-f316-4f9c-9a5d-01db442dcf74" (UID: "592eef38-f316-4f9c-9a5d-01db442dcf74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:42.918753 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:42.918731    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592eef38-f316-4f9c-9a5d-01db442dcf74-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:42.920286 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:42.920259    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592eef38-f316-4f9c-9a5d-01db442dcf74-kube-api-access-r89f2" pod "592eef38-f316-4f9c-9a5d-01db442dcf74" (UID: "592eef38-f316-4f9c-9a5d-01db442dcf74"). InnerVolumeSpecName "kube-api-access-r89f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:43.020047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.019953    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6baaf7a-c642-4575-84a4-7ad1035bcce6-kube-api-access-7p8bc\" (UniqueName: \"kubernetes.io/projected/a6baaf7a-c642-4575-84a4-7ad1035bcce6-kube-api-access-7p8bc\") pod \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\" (UID: \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\") "
Nov 02 02:05:43.020249 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.020137    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6baaf7a-c642-4575-84a4-7ad1035bcce6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6baaf7a-c642-4575-84a4-7ad1035bcce6-plugins-config-volume\") pod \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\" (UID: \"a6baaf7a-c642-4575-84a4-7ad1035bcce6\") "
Nov 02 02:05:43.020579 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.020551    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6baaf7a-c642-4575-84a4-7ad1035bcce6-plugins-config-volume" pod "a6baaf7a-c642-4575-84a4-7ad1035bcce6" (UID: "a6baaf7a-c642-4575-84a4-7ad1035bcce6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:43.020961 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.020940    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r89f2\" (UniqueName: \"kubernetes.io/projected/592eef38-f316-4f9c-9a5d-01db442dcf74-kube-api-access-r89f2\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:43.021005 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.020970    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6baaf7a-c642-4575-84a4-7ad1035bcce6-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:43.022301 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.022277    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6baaf7a-c642-4575-84a4-7ad1035bcce6-kube-api-access-7p8bc" pod "a6baaf7a-c642-4575-84a4-7ad1035bcce6" (UID: "a6baaf7a-c642-4575-84a4-7ad1035bcce6"). InnerVolumeSpecName "kube-api-access-7p8bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:43.091352 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.091320    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:05:43.122471 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.122440    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8bc\" (UniqueName: \"kubernetes.io/projected/a6baaf7a-c642-4575-84a4-7ad1035bcce6-kube-api-access-7p8bc\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:43.201967 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.201932    4912 generic.go:356] "Generic (PLEG): container finished" podID="592eef38-f316-4f9c-9a5d-01db442dcf74" containerID="495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7" exitCode=0
Nov 02 02:05:43.202473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.201985    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"
Nov 02 02:05:43.202473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.202015    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb" event={"ID":"592eef38-f316-4f9c-9a5d-01db442dcf74","Type":"ContainerDied","Data":"495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7"}
Nov 02 02:05:43.202473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.202119    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb" event={"ID":"592eef38-f316-4f9c-9a5d-01db442dcf74","Type":"ContainerDied","Data":"b138f926a31bded0c4bc389f90ce3713c10b051db5915c3b314db72cc851b1f6"}
Nov 02 02:05:43.202473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.202158    4912 scope.go:122] "RemoveContainer" containerID="495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7"
Nov 02 02:05:43.203778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.203755    4912 generic.go:356] "Generic (PLEG): container finished" podID="a6baaf7a-c642-4575-84a4-7ad1035bcce6" containerID="7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb" exitCode=0
Nov 02 02:05:43.204117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.203788    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z" event={"ID":"a6baaf7a-c642-4575-84a4-7ad1035bcce6","Type":"ContainerDied","Data":"7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb"}
Nov 02 02:05:43.204117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.203807    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"
Nov 02 02:05:43.204117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.203849    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z" event={"ID":"a6baaf7a-c642-4575-84a4-7ad1035bcce6","Type":"ContainerDied","Data":"1ca6ab378b4620714f31de7dab634ddf82c3c4c75e7974f5499d763da18b4de1"}
Nov 02 02:05:43.215590 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.215566    4912 scope.go:122] "RemoveContainer" containerID="495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7"
Nov 02 02:05:43.215891 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:05:43.215870    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7\": container with ID starting with 495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7 not found: ID does not exist" containerID="495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7"
Nov 02 02:05:43.215964 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.215898    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7"} err="failed to get container status \"495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7\": rpc error: code = NotFound desc = could not find container \"495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7\": container with ID starting with 495106b9325ccd35fbf3895784dd014c001f38b66dbedafeaad5ee8f64d3a2e7 not found: ID does not exist"
Nov 02 02:05:43.215964 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.215918    4912 scope.go:122] "RemoveContainer" containerID="7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb"
Nov 02 02:05:43.229999 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.229978    4912 scope.go:122] "RemoveContainer" containerID="7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb"
Nov 02 02:05:43.230366 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:05:43.230340    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb\": container with ID starting with 7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb not found: ID does not exist" containerID="7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb"
Nov 02 02:05:43.230474 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.230374    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb"} err="failed to get container status \"7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb\": rpc error: code = NotFound desc = could not find container \"7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb\": container with ID starting with 7e0328e6ff3587db6c0af04c039d22715ebe3a999f6d6e38f1c694326933c5fb not found: ID does not exist"
Nov 02 02:05:43.426581 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.426534    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"]
Nov 02 02:05:43.431926 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.431886    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-ghxgb"]
Nov 02 02:05:43.541120 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.541084    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"]
Nov 02 02:05:43.546637 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.546604    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-hbj6z"]
Nov 02 02:05:43.719584 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:43.719490    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:45.227369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:45.227335    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592eef38-f316-4f9c-9a5d-01db442dcf74" path="/var/lib/kubelet/pods/592eef38-f316-4f9c-9a5d-01db442dcf74/volumes"
Nov 02 02:05:45.227776 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:45.227766    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6baaf7a-c642-4575-84a4-7ad1035bcce6" path="/var/lib/kubelet/pods/a6baaf7a-c642-4575-84a4-7ad1035bcce6/volumes"
Nov 02 02:05:46.527778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.527744    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"]
Nov 02 02:05:46.528582 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.528562    4912 cpu_manager.go:414] "containerMap: removing container" podUID="592eef38-f316-4f9c-9a5d-01db442dcf74" containerName="payload-processing"
Nov 02 02:05:46.528662 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.528584    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592eef38-f316-4f9c-9a5d-01db442dcf74" containerName="payload-processing"
Nov 02 02:05:46.528662 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.528610    4912 cpu_manager.go:414] "containerMap: removing container" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy"
Nov 02 02:05:46.528662 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.528615    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy"
Nov 02 02:05:46.528662 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.528633    4912 cpu_manager.go:414] "containerMap: removing container" podUID="a6baaf7a-c642-4575-84a4-7ad1035bcce6" containerName="payload-pre-processing"
Nov 02 02:05:46.528662 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.528638    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6baaf7a-c642-4575-84a4-7ad1035bcce6" containerName="payload-pre-processing"
Nov 02 02:05:46.528843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.528796    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6baaf7a-c642-4575-84a4-7ad1035bcce6" containerName="payload-pre-processing"
Nov 02 02:05:46.528843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.528814    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="592eef38-f316-4f9c-9a5d-01db442dcf74" containerName="payload-processing"
Nov 02 02:05:46.528843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.528823    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="94764ea2-8bb7-4ce7-897a-fd0a1886f97f" containerName="istio-proxy"
Nov 02 02:05:46.534370 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.534347    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.575996 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.575966    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.576234 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.576212    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.576326 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.576262    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjllh\" (UniqueName: \"kubernetes.io/projected/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-kube-api-access-cjllh\") pod \"maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.589163 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.589141    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"]
Nov 02 02:05:46.632844 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.632798    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:05:46.633126 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.632890    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:05:46.678166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.678134    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjllh\" (UniqueName: \"kubernetes.io/projected/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-kube-api-access-cjllh\") pod \"maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.678382 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.678360    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.678564 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.678547    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.679199 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.679175    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.681184 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.681163    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.739301 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.739268    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjllh\" (UniqueName: \"kubernetes.io/projected/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-kube-api-access-cjllh\") pod \"maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:46.850774 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:46.850696    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:47.242192 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:47.242159    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"]
Nov 02 02:05:48.257784 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:48.257733    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt" event={"ID":"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c","Type":"ContainerStarted","Data":"45b51ebd55fbf66e988ced770ca8374b6723fd39f2cd4ef3382ac9aa6192a314"}
Nov 02 02:05:48.258233 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:48.257802    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt" event={"ID":"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c","Type":"ContainerStarted","Data":"5b942b3a400d1d7c3c201f7ae93c3a0bb3e1a0a7919ff0a8ceade787588b5a32"}
Nov 02 02:05:48.372962 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:48.372893    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt" podStartSLOduration=2.372876275 podStartE2EDuration="2.372876275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:05:48.372205623 +0000 UTC m=+2041.277670980" watchObservedRunningTime="2026-11-02 02:05:48.372876275 +0000 UTC m=+2041.278341633"
Nov 02 02:05:48.571099 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:48.570977    4912 scope.go:122] "RemoveContainer" containerID="0c887701231d8e61ddfd527b0360d0d8f427678a41a5a781f021fb180dc61129"
Nov 02 02:05:48.778259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:48.778196    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:49.269457 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:49.269417    4912 generic.go:356] "Generic (PLEG): container finished" podID="bdbcf169-58cd-40a7-83e0-103a2e1c8f6c" containerID="45b51ebd55fbf66e988ced770ca8374b6723fd39f2cd4ef3382ac9aa6192a314" exitCode=0
Nov 02 02:05:49.269925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:49.269499    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt" event={"ID":"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c","Type":"ContainerDied","Data":"45b51ebd55fbf66e988ced770ca8374b6723fd39f2cd4ef3382ac9aa6192a314"}
Nov 02 02:05:49.445834 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:49.445798    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:50.462406 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.462377    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:50.537020 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.536928    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-xks-ca\") pod \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") "
Nov 02 02:05:50.537226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.537108    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-openshift-ca\") pod \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") "
Nov 02 02:05:50.537226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.537200    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-kube-api-access-cjllh\" (UniqueName: \"kubernetes.io/projected/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-kube-api-access-cjllh\") pod \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\" (UID: \"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c\") "
Nov 02 02:05:50.537539 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.537512    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-openshift-ca" pod "bdbcf169-58cd-40a7-83e0-103a2e1c8f6c" (UID: "bdbcf169-58cd-40a7-83e0-103a2e1c8f6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:50.538293 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.538264    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:50.539765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.539733    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-xks-ca" pod "bdbcf169-58cd-40a7-83e0-103a2e1c8f6c" (UID: "bdbcf169-58cd-40a7-83e0-103a2e1c8f6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:05:50.539867 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.539779    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-kube-api-access-cjllh" pod "bdbcf169-58cd-40a7-83e0-103a2e1c8f6c" (UID: "bdbcf169-58cd-40a7-83e0-103a2e1c8f6c"). InnerVolumeSpecName "kube-api-access-cjllh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:50.640620 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.640578    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjllh\" (UniqueName: \"kubernetes.io/projected/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-kube-api-access-cjllh\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:50.640620 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:50.640624    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.289616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:51.289581    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"
Nov 02 02:05:51.289768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:51.289620    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt" event={"ID":"bdbcf169-58cd-40a7-83e0-103a2e1c8f6c","Type":"ContainerDied","Data":"5b942b3a400d1d7c3c201f7ae93c3a0bb3e1a0a7919ff0a8ceade787588b5a32"}
Nov 02 02:05:51.289768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:51.289671    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b942b3a400d1d7c3c201f7ae93c3a0bb3e1a0a7919ff0a8ceade787588b5a32"
Nov 02 02:05:52.805276 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:52.805236    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:53.039657 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:53.039619    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:53.576358 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:53.576318    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:58.596842 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:58.596788    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:58.827617 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:58.827565    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:05:59.011685 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:05:59.011651    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:03.791798 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:03.791758    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:07.108641 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:07.108605    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"]
Nov 02 02:06:07.109185 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:07.108858    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy" containerID="cri-o://233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca" gracePeriod=30
Nov 02 02:06:08.518821 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:08.518786    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:09.897284 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:09.897235    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"]
Nov 02 02:06:09.898199 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:09.898177    4912 cpu_manager.go:414] "containerMap: removing container" podUID="bdbcf169-58cd-40a7-83e0-103a2e1c8f6c" containerName="revoke-keys"
Nov 02 02:06:09.898199 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:09.898199    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbcf169-58cd-40a7-83e0-103a2e1c8f6c" containerName="revoke-keys"
Nov 02 02:06:09.898476 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:09.898460    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbcf169-58cd-40a7-83e0-103a2e1c8f6c" containerName="revoke-keys"
Nov 02 02:06:09.902261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:09.902239    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:09.958891 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:09.958858    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"]
Nov 02 02:06:09.992461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:09.992417    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:09.992639 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:09.992507    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t85rn\" (UniqueName: \"kubernetes.io/projected/d66d8b4d-fa84-4a75-b850-e2197402a346-kube-api-access-t85rn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:09.992779 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:09.992750    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:10.003345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.003312    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:06:10.003554 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.003367    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:06:10.093718 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.093683    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t85rn\" (UniqueName: \"kubernetes.io/projected/d66d8b4d-fa84-4a75-b850-e2197402a346-kube-api-access-t85rn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:10.093938 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.093840    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:10.094044 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.094025    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:10.094887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.094848    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:10.097094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.097044    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:10.154936 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.154853    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t85rn\" (UniqueName: \"kubernetes.io/projected/d66d8b4d-fa84-4a75-b850-e2197402a346-kube-api-access-t85rn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:10.218265 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.218232    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:10.613944 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:06:10.613910    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66d8b4d_fa84_4a75_b850_e2197402a346.slice/crio-e81e355daa76e0a390d705f39312dc7abbaf45cb2a9ebb70f98ee35e9533a7e8 WatchSource:0}: Error finding container e81e355daa76e0a390d705f39312dc7abbaf45cb2a9ebb70f98ee35e9533a7e8: Status 404 returned error can't find the container with id e81e355daa76e0a390d705f39312dc7abbaf45cb2a9ebb70f98ee35e9533a7e8
Nov 02 02:06:10.614335 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:10.614308    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"]
Nov 02 02:06:11.531228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:11.531186    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv" event={"ID":"d66d8b4d-fa84-4a75-b850-e2197402a346","Type":"ContainerStarted","Data":"8b4236c21a8e2ceb479421f260ee58da5f8cb4a7fcaa492229ec7876c2ed2571"}
Nov 02 02:06:11.531703 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:11.531267    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv" event={"ID":"d66d8b4d-fa84-4a75-b850-e2197402a346","Type":"ContainerStarted","Data":"e81e355daa76e0a390d705f39312dc7abbaf45cb2a9ebb70f98ee35e9533a7e8"}
Nov 02 02:06:11.650849 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:11.650790    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv" podStartSLOduration=2.650773253 podStartE2EDuration="2.650773253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:11.646323849 +0000 UTC m=+2064.551789206" watchObservedRunningTime="2026-11-02 02:06:11.650773253 +0000 UTC m=+2064.556238611"
Nov 02 02:06:12.142913 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:06:12.142880    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:06:12.397543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.397482    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 02:06:12.537952 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.537924    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-kube-api-access-8zbsn\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-kube-api-access-8zbsn\") pod \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") "
Nov 02 02:06:12.538519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.537962    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-socket\") pod \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") "
Nov 02 02:06:12.538519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.537982    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-credential-socket\") pod \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") "
Nov 02 02:06:12.538519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.538002    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-token\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-token\") pod \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") "
Nov 02 02:06:12.538519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.538227    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istiod-ca-cert\") pod \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") "
Nov 02 02:06:12.538519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.538262    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-envoy\") pod \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") "
Nov 02 02:06:12.538519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.538289    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-certs\") pod \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") "
Nov 02 02:06:12.538519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.538372    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-socket" pod "7bbfe2bb-a981-4339-8a0c-aa7d77994be0" (UID: "7bbfe2bb-a981-4339-8a0c-aa7d77994be0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:12.538519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.538458    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-podinfo\") pod \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") "
Nov 02 02:06:12.538972 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.538526    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-data\") pod \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\" (UID: \"7bbfe2bb-a981-4339-8a0c-aa7d77994be0\") "
Nov 02 02:06:12.538972 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.538640    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-certs" pod "7bbfe2bb-a981-4339-8a0c-aa7d77994be0" (UID: "7bbfe2bb-a981-4339-8a0c-aa7d77994be0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:12.539276 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.539012    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istiod-ca-cert" pod "7bbfe2bb-a981-4339-8a0c-aa7d77994be0" (UID: "7bbfe2bb-a981-4339-8a0c-aa7d77994be0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:12.539276 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.539260    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:12.539387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.539281    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:12.539387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.539296    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:12.539387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.539311    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-credential-socket" pod "7bbfe2bb-a981-4339-8a0c-aa7d77994be0" (UID: "7bbfe2bb-a981-4339-8a0c-aa7d77994be0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:12.539387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.539354    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-data" pod "7bbfe2bb-a981-4339-8a0c-aa7d77994be0" (UID: "7bbfe2bb-a981-4339-8a0c-aa7d77994be0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:12.541232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.541193    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-token" pod "7bbfe2bb-a981-4339-8a0c-aa7d77994be0" (UID: "7bbfe2bb-a981-4339-8a0c-aa7d77994be0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:12.541356 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.541237    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-envoy" pod "7bbfe2bb-a981-4339-8a0c-aa7d77994be0" (UID: "7bbfe2bb-a981-4339-8a0c-aa7d77994be0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:12.541356 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.541306    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-podinfo" pod "7bbfe2bb-a981-4339-8a0c-aa7d77994be0" (UID: "7bbfe2bb-a981-4339-8a0c-aa7d77994be0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:06:12.542592 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.542570    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-kube-api-access-8zbsn" pod "7bbfe2bb-a981-4339-8a0c-aa7d77994be0" (UID: "7bbfe2bb-a981-4339-8a0c-aa7d77994be0"). InnerVolumeSpecName "kube-api-access-8zbsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:12.543608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.543579    4912 generic.go:356] "Generic (PLEG): container finished" podID="d66d8b4d-fa84-4a75-b850-e2197402a346" containerID="8b4236c21a8e2ceb479421f260ee58da5f8cb4a7fcaa492229ec7876c2ed2571" exitCode=0
Nov 02 02:06:12.543770 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.543751    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv" event={"ID":"d66d8b4d-fa84-4a75-b850-e2197402a346","Type":"ContainerDied","Data":"8b4236c21a8e2ceb479421f260ee58da5f8cb4a7fcaa492229ec7876c2ed2571"}
Nov 02 02:06:12.545384 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.545361    4912 generic.go:356] "Generic (PLEG): container finished" podID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerID="233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca" exitCode=0
Nov 02 02:06:12.545505 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.545420    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"
Nov 02 02:06:12.545505 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.545448    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" event={"ID":"7bbfe2bb-a981-4339-8a0c-aa7d77994be0","Type":"ContainerDied","Data":"233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca"}
Nov 02 02:06:12.545633 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.545507    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc" event={"ID":"7bbfe2bb-a981-4339-8a0c-aa7d77994be0","Type":"ContainerDied","Data":"9aabf4890eb624b30d1d18b525ef6afb324ce09760bf90d18811d46a583d380f"}
Nov 02 02:06:12.545633 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.545538    4912 scope.go:122] "RemoveContainer" containerID="233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca"
Nov 02 02:06:12.590497 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.590469    4912 scope.go:122] "RemoveContainer" containerID="233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca"
Nov 02 02:06:12.590856 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:06:12.590836    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca\": container with ID starting with 233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca not found: ID does not exist" containerID="233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca"
Nov 02 02:06:12.590931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.590865    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca"} err="failed to get container status \"233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca\": rpc error: code = NotFound desc = could not find container \"233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca\": container with ID starting with 233ce9ffa3e1452383d6370c851a30103e5c5f0a1425a6ae24eb6809348e2dca not found: ID does not exist"
Nov 02 02:06:12.640285 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.640254    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:12.640285 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.640283    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:12.640510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.640294    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbsn\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-kube-api-access-8zbsn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:12.640510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.640305    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:12.640510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.640313    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:12.640510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.640322    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbfe2bb-a981-4339-8a0c-aa7d77994be0-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:12.651523 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:06:12.651432    4912 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbfe2bb_a981_4339_8a0c_aa7d77994be0.slice/crio-9aabf4890eb624b30d1d18b525ef6afb324ce09760bf90d18811d46a583d380f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbfe2bb_a981_4339_8a0c_aa7d77994be0.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 02:06:12.862695 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.862656    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"]
Nov 02 02:06:12.867209 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:12.867185    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-c66qc"]
Nov 02 02:06:13.228137 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.228102    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" path="/var/lib/kubelet/pods/7bbfe2bb-a981-4339-8a0c-aa7d77994be0/volumes"
Nov 02 02:06:13.705304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.705257    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:13.799386 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.799357    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:13.962513 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.962474    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d66d8b4d-fa84-4a75-b850-e2197402a346-kube-api-access-t85rn\" (UniqueName: \"kubernetes.io/projected/d66d8b4d-fa84-4a75-b850-e2197402a346-kube-api-access-t85rn\") pod \"d66d8b4d-fa84-4a75-b850-e2197402a346\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") "
Nov 02 02:06:13.962735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.962716    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-openshift-ca\") pod \"d66d8b4d-fa84-4a75-b850-e2197402a346\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") "
Nov 02 02:06:13.962805 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.962768    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-xks-ca\") pod \"d66d8b4d-fa84-4a75-b850-e2197402a346\" (UID: \"d66d8b4d-fa84-4a75-b850-e2197402a346\") "
Nov 02 02:06:13.963205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.963174    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-openshift-ca" pod "d66d8b4d-fa84-4a75-b850-e2197402a346" (UID: "d66d8b4d-fa84-4a75-b850-e2197402a346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:13.963538 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.963511    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:13.965087 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.965042    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66d8b4d-fa84-4a75-b850-e2197402a346-kube-api-access-t85rn" pod "d66d8b4d-fa84-4a75-b850-e2197402a346" (UID: "d66d8b4d-fa84-4a75-b850-e2197402a346"). InnerVolumeSpecName "kube-api-access-t85rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:13.965532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:13.965511    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-xks-ca" pod "d66d8b4d-fa84-4a75-b850-e2197402a346" (UID: "d66d8b4d-fa84-4a75-b850-e2197402a346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:14.065555 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:14.065520    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66d8b4d-fa84-4a75-b850-e2197402a346-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:14.065555 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:14.065550    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t85rn\" (UniqueName: \"kubernetes.io/projected/d66d8b4d-fa84-4a75-b850-e2197402a346-kube-api-access-t85rn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:14.570650 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:14.570602    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv" event={"ID":"d66d8b4d-fa84-4a75-b850-e2197402a346","Type":"ContainerDied","Data":"e81e355daa76e0a390d705f39312dc7abbaf45cb2a9ebb70f98ee35e9533a7e8"}
Nov 02 02:06:14.570843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:14.570711    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e81e355daa76e0a390d705f39312dc7abbaf45cb2a9ebb70f98ee35e9533a7e8"
Nov 02 02:06:14.570843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:14.570836    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"
Nov 02 02:06:15.911089 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:15.911032    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"]
Nov 02 02:06:15.911589 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:15.911301    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy" containerID="cri-o://542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315" gracePeriod=30
Nov 02 02:06:15.963607 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:15.963569    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"]
Nov 02 02:06:15.963846 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:15.963820    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy" containerID="cri-o://b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58" gracePeriod=30
Nov 02 02:06:18.062811 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:18.062774    4912 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:06:18.063357 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:18.062850    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:06:18.859187 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:18.859143    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:21.286128 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.283715    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 02:06:21.322855 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.322827    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 02:06:21.380021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.379988    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532f4fe9-2bb4-4324-9951-976238151431-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f4fe9-2bb4-4324-9951-976238151431-istiod-ca-cert\") pod \"532f4fe9-2bb4-4324-9951-976238151431\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") "
Nov 02 02:06:21.380269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380041    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-envoy\") pod \"532f4fe9-2bb4-4324-9951-976238151431\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") "
Nov 02 02:06:21.380269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380097    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-kube-api-access-7lqkr\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-kube-api-access-7lqkr\") pod \"532f4fe9-2bb4-4324-9951-976238151431\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") "
Nov 02 02:06:21.380269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380187    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-podinfo\") pod \"a16fb43b-1c92-40fb-935b-a25a511b71d1\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") "
Nov 02 02:06:21.380269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380235    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-istio-token\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-istio-token\") pod \"532f4fe9-2bb4-4324-9951-976238151431\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") "
Nov 02 02:06:21.380501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380298    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-credential-socket\") pod \"a16fb43b-1c92-40fb-935b-a25a511b71d1\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") "
Nov 02 02:06:21.380501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380347    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-socket\") pod \"532f4fe9-2bb4-4324-9951-976238151431\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") "
Nov 02 02:06:21.380501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380416    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/532f4fe9-2bb4-4324-9951-976238151431-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f4fe9-2bb4-4324-9951-976238151431-istio-podinfo\") pod \"532f4fe9-2bb4-4324-9951-976238151431\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") "
Nov 02 02:06:21.380501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380451    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-envoy\") pod \"a16fb43b-1c92-40fb-935b-a25a511b71d1\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") "
Nov 02 02:06:21.380501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380484    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532f4fe9-2bb4-4324-9951-976238151431-istiod-ca-cert" pod "532f4fe9-2bb4-4324-9951-976238151431" (UID: "532f4fe9-2bb4-4324-9951-976238151431"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:21.380881 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380536    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-token\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-token\") pod \"a16fb43b-1c92-40fb-935b-a25a511b71d1\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") "
Nov 02 02:06:21.380881 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380584    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-data\") pod \"532f4fe9-2bb4-4324-9951-976238151431\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") "
Nov 02 02:06:21.380881 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380620    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-certs\") pod \"a16fb43b-1c92-40fb-935b-a25a511b71d1\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") "
Nov 02 02:06:21.380881 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380648    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16fb43b-1c92-40fb-935b-a25a511b71d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16fb43b-1c92-40fb-935b-a25a511b71d1-istiod-ca-cert\") pod \"a16fb43b-1c92-40fb-935b-a25a511b71d1\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") "
Nov 02 02:06:21.380881 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380702    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-credential-socket\") pod \"532f4fe9-2bb4-4324-9951-976238151431\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") "
Nov 02 02:06:21.380881 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380749    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-socket\") pod \"a16fb43b-1c92-40fb-935b-a25a511b71d1\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") "
Nov 02 02:06:21.380881 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380777    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-kube-api-access-6czb8\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-kube-api-access-6czb8\") pod \"a16fb43b-1c92-40fb-935b-a25a511b71d1\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") "
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.380819    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-data\") pod \"a16fb43b-1c92-40fb-935b-a25a511b71d1\" (UID: \"a16fb43b-1c92-40fb-935b-a25a511b71d1\") "
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.381150    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-certs\") pod \"532f4fe9-2bb4-4324-9951-976238151431\" (UID: \"532f4fe9-2bb4-4324-9951-976238151431\") "
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.381731    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-credential-socket" pod "a16fb43b-1c92-40fb-935b-a25a511b71d1" (UID: "a16fb43b-1c92-40fb-935b-a25a511b71d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.381953    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-socket" pod "a16fb43b-1c92-40fb-935b-a25a511b71d1" (UID: "a16fb43b-1c92-40fb-935b-a25a511b71d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.382038    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-certs" pod "a16fb43b-1c92-40fb-935b-a25a511b71d1" (UID: "a16fb43b-1c92-40fb-935b-a25a511b71d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.382272    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16fb43b-1c92-40fb-935b-a25a511b71d1-istiod-ca-cert" pod "a16fb43b-1c92-40fb-935b-a25a511b71d1" (UID: "a16fb43b-1c92-40fb-935b-a25a511b71d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.382382    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-credential-socket" pod "532f4fe9-2bb4-4324-9951-976238151431" (UID: "532f4fe9-2bb4-4324-9951-976238151431"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.382612    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-socket" pod "532f4fe9-2bb4-4324-9951-976238151431" (UID: "532f4fe9-2bb4-4324-9951-976238151431"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383336    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-data" pod "532f4fe9-2bb4-4324-9951-976238151431" (UID: "532f4fe9-2bb4-4324-9951-976238151431"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.383412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383369    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-data" pod "a16fb43b-1c92-40fb-935b-a25a511b71d1" (UID: "a16fb43b-1c92-40fb-935b-a25a511b71d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383615    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-certs" pod "532f4fe9-2bb4-4324-9951-976238151431" (UID: "532f4fe9-2bb4-4324-9951-976238151431"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383846    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f4fe9-2bb4-4324-9951-976238151431-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383865    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383874    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383884    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383893    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383901    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16fb43b-1c92-40fb-935b-a25a511b71d1-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383912    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383926    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383936    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383933    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-kube-api-access-7lqkr" pod "532f4fe9-2bb4-4324-9951-976238151431" (UID: "532f4fe9-2bb4-4324-9951-976238151431"). InnerVolumeSpecName "kube-api-access-7lqkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:21.384008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.383945    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.385106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.385036    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-podinfo" pod "a16fb43b-1c92-40fb-935b-a25a511b71d1" (UID: "a16fb43b-1c92-40fb-935b-a25a511b71d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:06:21.385390 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.385361    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-envoy" pod "532f4fe9-2bb4-4324-9951-976238151431" (UID: "532f4fe9-2bb4-4324-9951-976238151431"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.385709 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.385683    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-istio-token" pod "532f4fe9-2bb4-4324-9951-976238151431" (UID: "532f4fe9-2bb4-4324-9951-976238151431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:21.385781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.385687    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-envoy" pod "a16fb43b-1c92-40fb-935b-a25a511b71d1" (UID: "a16fb43b-1c92-40fb-935b-a25a511b71d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:21.385781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.385756    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/532f4fe9-2bb4-4324-9951-976238151431-istio-podinfo" pod "532f4fe9-2bb4-4324-9951-976238151431" (UID: "532f4fe9-2bb4-4324-9951-976238151431"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:06:21.386636 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.386617    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-token" pod "a16fb43b-1c92-40fb-935b-a25a511b71d1" (UID: "a16fb43b-1c92-40fb-935b-a25a511b71d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:21.387460 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.387442    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-kube-api-access-6czb8" pod "a16fb43b-1c92-40fb-935b-a25a511b71d1" (UID: "a16fb43b-1c92-40fb-935b-a25a511b71d1"). InnerVolumeSpecName "kube-api-access-6czb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:21.400709 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.400677    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"]
Nov 02 02:06:21.485047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.484963    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6czb8\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-kube-api-access-6czb8\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.485047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.485000    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f4fe9-2bb4-4324-9951-976238151431-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.485047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.485011    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lqkr\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-kube-api-access-7lqkr\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.485047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.485019    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.485047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.485032    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532f4fe9-2bb4-4324-9951-976238151431-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.485047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.485044    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f4fe9-2bb4-4324-9951-976238151431-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.485047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.485052    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.485047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.485074    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16fb43b-1c92-40fb-935b-a25a511b71d1-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:21.513628 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.513594    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"]
Nov 02 02:06:21.636108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.636073    4912 generic.go:356] "Generic (PLEG): container finished" podID="532f4fe9-2bb4-4324-9951-976238151431" containerID="542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315" exitCode=0
Nov 02 02:06:21.636304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.636151    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"
Nov 02 02:06:21.636304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.636149    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" event={"ID":"532f4fe9-2bb4-4324-9951-976238151431","Type":"ContainerDied","Data":"542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315"}
Nov 02 02:06:21.636304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.636280    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95" event={"ID":"532f4fe9-2bb4-4324-9951-976238151431","Type":"ContainerDied","Data":"76bf89a5a913beb1ec1151b60fda47a5d695275f1a24b0cfe668ff7afbc594b9"}
Nov 02 02:06:21.636481 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.636314    4912 scope.go:122] "RemoveContainer" containerID="542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315"
Nov 02 02:06:21.643155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.642169    4912 generic.go:356] "Generic (PLEG): container finished" podID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerID="b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58" exitCode=0
Nov 02 02:06:21.643155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.642268    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" event={"ID":"a16fb43b-1c92-40fb-935b-a25a511b71d1","Type":"ContainerDied","Data":"b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58"}
Nov 02 02:06:21.643155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.642502    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"
Nov 02 02:06:21.648184 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.648147    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt" event={"ID":"a16fb43b-1c92-40fb-935b-a25a511b71d1","Type":"ContainerDied","Data":"f0e4d48cbfcd7afe4f2d8e4f0913bcca0af1735062d2ecf7663dd14fe558887f"}
Nov 02 02:06:21.663255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.663228    4912 scope.go:122] "RemoveContainer" containerID="542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315"
Nov 02 02:06:21.663565 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:06:21.663536    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315\": container with ID starting with 542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315 not found: ID does not exist" containerID="542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315"
Nov 02 02:06:21.663641 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.663578    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315"} err="failed to get container status \"542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315\": rpc error: code = NotFound desc = could not find container \"542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315\": container with ID starting with 542a584f389d4cbcae4817c856418785fbbf092bdecd02fbdbbc3201186b8315 not found: ID does not exist"
Nov 02 02:06:21.663641 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.663600    4912 scope.go:122] "RemoveContainer" containerID="b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58"
Nov 02 02:06:21.674689 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.674660    4912 scope.go:122] "RemoveContainer" containerID="b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58"
Nov 02 02:06:21.675035 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:06:21.675007    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58\": container with ID starting with b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58 not found: ID does not exist" containerID="b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58"
Nov 02 02:06:21.675172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.675041    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58"} err="failed to get container status \"b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58\": rpc error: code = NotFound desc = could not find container \"b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58\": container with ID starting with b0c4321a66bb80cbf1017d35106b5436b340bc915dee61540808a7bd908a7e58 not found: ID does not exist"
Nov 02 02:06:21.796313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.796237    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:21.805265 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.805236    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"]
Nov 02 02:06:21.818308 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.814429    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-wlq95"]
Nov 02 02:06:21.927570 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.927517    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"]
Nov 02 02:06:21.934035 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.934000    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-nxzbt"]
Nov 02 02:06:21.981857 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:21.981824    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:23.228013 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.227980    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532f4fe9-2bb4-4324-9951-976238151431" path="/var/lib/kubelet/pods/532f4fe9-2bb4-4324-9951-976238151431/volumes"
Nov 02 02:06:23.228522 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.228507    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" path="/var/lib/kubelet/pods/a16fb43b-1c92-40fb-935b-a25a511b71d1/volumes"
Nov 02 02:06:23.378314 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.378277    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"]
Nov 02 02:06:23.378499 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.378478    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7" podUID="1fe1d8e1-10e5-4225-bcbe-66653abe617f" containerName="discovery" containerID="cri-o://e79f137379ef127c1832955bb0a6937f3f3a17bef9802b93ae7fcc7d7ef5d961" gracePeriod=30
Nov 02 02:06:23.641406 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.641367    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:23.710301 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.703458    4912 generic.go:356] "Generic (PLEG): container finished" podID="1fe1d8e1-10e5-4225-bcbe-66653abe617f" containerID="e79f137379ef127c1832955bb0a6937f3f3a17bef9802b93ae7fcc7d7ef5d961" exitCode=0
Nov 02 02:06:23.710301 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.703668    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7" event={"ID":"1fe1d8e1-10e5-4225-bcbe-66653abe617f","Type":"ContainerDied","Data":"e79f137379ef127c1832955bb0a6937f3f3a17bef9802b93ae7fcc7d7ef5d961"}
Nov 02 02:06:23.782486 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.782429    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 02:06:23.842141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.842106    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-ca-configmap\") pod \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") "
Nov 02 02:06:23.842304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.842175    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-cacerts\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-cacerts\") pod \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") "
Nov 02 02:06:23.842369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.842322    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-token\") pod \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") "
Nov 02 02:06:23.842369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.842360    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-dns-cert\") pod \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") "
Nov 02 02:06:23.842483 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.842405    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-kube-api-access-c5rvt\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-kube-api-access-c5rvt\") pod \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") "
Nov 02 02:06:23.842545 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.842488    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-kubeconfig\") pod \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") "
Nov 02 02:06:23.842545 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.842500    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-ca-configmap" pod "1fe1d8e1-10e5-4225-bcbe-66653abe617f" (UID: "1fe1d8e1-10e5-4225-bcbe-66653abe617f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:23.842545 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.842524    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1d8e1-10e5-4225-bcbe-66653abe617f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1d8e1-10e5-4225-bcbe-66653abe617f-local-certs\") pod \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\" (UID: \"1fe1d8e1-10e5-4225-bcbe-66653abe617f\") "
Nov 02 02:06:23.843414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.843384    4912 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-ca-configmap\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:23.844945 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.844921    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1d8e1-10e5-4225-bcbe-66653abe617f-local-certs" pod "1fe1d8e1-10e5-4225-bcbe-66653abe617f" (UID: "1fe1d8e1-10e5-4225-bcbe-66653abe617f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:23.845088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.844985    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-dns-cert" pod "1fe1d8e1-10e5-4225-bcbe-66653abe617f" (UID: "1fe1d8e1-10e5-4225-bcbe-66653abe617f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:23.845289 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.845271    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-token" pod "1fe1d8e1-10e5-4225-bcbe-66653abe617f" (UID: "1fe1d8e1-10e5-4225-bcbe-66653abe617f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:23.845401 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.845379    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-cacerts" pod "1fe1d8e1-10e5-4225-bcbe-66653abe617f" (UID: "1fe1d8e1-10e5-4225-bcbe-66653abe617f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:23.845444 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.845397    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-kubeconfig" pod "1fe1d8e1-10e5-4225-bcbe-66653abe617f" (UID: "1fe1d8e1-10e5-4225-bcbe-66653abe617f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:23.845444 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.845427    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-kube-api-access-c5rvt" pod "1fe1d8e1-10e5-4225-bcbe-66653abe617f" (UID: "1fe1d8e1-10e5-4225-bcbe-66653abe617f"). InnerVolumeSpecName "kube-api-access-c5rvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:23.945165 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.945125    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:23.945165 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.945156    4912 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-csr-dns-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:23.945165 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.945167    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5rvt\" (UniqueName: \"kubernetes.io/projected/1fe1d8e1-10e5-4225-bcbe-66653abe617f-kube-api-access-c5rvt\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:23.945485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.945177    4912 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-istio-kubeconfig\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:23.945485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.945186    4912 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1d8e1-10e5-4225-bcbe-66653abe617f-local-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:23.945485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:23.945195    4912 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1fe1d8e1-10e5-4225-bcbe-66653abe617f-cacerts\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:24.714282 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:24.714255    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"
Nov 02 02:06:24.714787 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:24.714283    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7" event={"ID":"1fe1d8e1-10e5-4225-bcbe-66653abe617f","Type":"ContainerDied","Data":"bc74b05f6c87a3015efa8f3fa68bf9a9aedbca818c17cfcfddec076a22084a7c"}
Nov 02 02:06:24.714787 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:24.714349    4912 scope.go:122] "RemoveContainer" containerID="e79f137379ef127c1832955bb0a6937f3f3a17bef9802b93ae7fcc7d7ef5d961"
Nov 02 02:06:24.881428 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:24.881393    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"]
Nov 02 02:06:24.930907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:24.930868    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2gcp7"]
Nov 02 02:06:25.226860 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:25.226826    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe1d8e1-10e5-4225-bcbe-66653abe617f" path="/var/lib/kubelet/pods/1fe1d8e1-10e5-4225-bcbe-66653abe617f/volumes"
Nov 02 02:06:26.326425 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.326390    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"]
Nov 02 02:06:26.327135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327110    4912 cpu_manager.go:414] "containerMap: removing container" podUID="1fe1d8e1-10e5-4225-bcbe-66653abe617f" containerName="discovery"
Nov 02 02:06:26.327135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327126    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe1d8e1-10e5-4225-bcbe-66653abe617f" containerName="discovery"
Nov 02 02:06:26.327240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327138    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d66d8b4d-fa84-4a75-b850-e2197402a346" containerName="revoke-keys"
Nov 02 02:06:26.327240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327144    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66d8b4d-fa84-4a75-b850-e2197402a346" containerName="revoke-keys"
Nov 02 02:06:26.327240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327161    4912 cpu_manager.go:414] "containerMap: removing container" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy"
Nov 02 02:06:26.327240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327166    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy"
Nov 02 02:06:26.327240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327177    4912 cpu_manager.go:414] "containerMap: removing container" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy"
Nov 02 02:06:26.327240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327181    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy"
Nov 02 02:06:26.327240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327191    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy"
Nov 02 02:06:26.327240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327196    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy"
Nov 02 02:06:26.327481 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327305    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbfe2bb-a981-4339-8a0c-aa7d77994be0" containerName="istio-proxy"
Nov 02 02:06:26.327481 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327321    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe1d8e1-10e5-4225-bcbe-66653abe617f" containerName="discovery"
Nov 02 02:06:26.327481 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327327    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66d8b4d-fa84-4a75-b850-e2197402a346" containerName="revoke-keys"
Nov 02 02:06:26.327481 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327336    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16fb43b-1c92-40fb-935b-a25a511b71d1" containerName="istio-proxy"
Nov 02 02:06:26.327481 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.327343    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="532f4fe9-2bb4-4324-9951-976238151431" containerName="istio-proxy"
Nov 02 02:06:26.332733 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.332714    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.390256 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.390218    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"]
Nov 02 02:06:26.401740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.401699    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr" podUID="62cd8a15-a27f-4c3f-aaa1-f8819bd6584f" containerName="payload-processing" containerID="cri-o://c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b" gracePeriod=25
Nov 02 02:06:26.432390 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.432341    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:06:26.432564 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.432476    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:06:26.481493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.481447    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmb4c\" (UniqueName: \"kubernetes.io/projected/f575054a-9c07-43be-83bd-867c03a71fe7-kube-api-access-cmb4c\") pod \"maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.481666 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.481623    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.481928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.481909    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.514172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.514101    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r" podUID="8167b26c-f67e-45fc-8691-95a15a9a6d92" containerName="payload-pre-processing" containerID="cri-o://96c3c6dbb8d37114a76793cb17a1734fd2dead00ce855a6010d1d16be69907c2" gracePeriod=25
Nov 02 02:06:26.589743 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.589708    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.589887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.589800    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmb4c\" (UniqueName: \"kubernetes.io/projected/f575054a-9c07-43be-83bd-867c03a71fe7-kube-api-access-cmb4c\") pod \"maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.589966 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.589882    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.591496 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.591394    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.604093 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.598765    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.662807 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.662766    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmb4c\" (UniqueName: \"kubernetes.io/projected/f575054a-9c07-43be-83bd-867c03a71fe7-kube-api-access-cmb4c\") pod \"maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.732088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.731772    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:06:26.738143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.736685    4912 generic.go:356] "Generic (PLEG): container finished" podID="8167b26c-f67e-45fc-8691-95a15a9a6d92" containerID="96c3c6dbb8d37114a76793cb17a1734fd2dead00ce855a6010d1d16be69907c2" exitCode=0
Nov 02 02:06:26.738143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.736931    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r" event={"ID":"8167b26c-f67e-45fc-8691-95a15a9a6d92","Type":"ContainerDied","Data":"96c3c6dbb8d37114a76793cb17a1734fd2dead00ce855a6010d1d16be69907c2"}
Nov 02 02:06:26.738704 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.738682    4912 generic.go:356] "Generic (PLEG): container finished" podID="62cd8a15-a27f-4c3f-aaa1-f8819bd6584f" containerID="c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b" exitCode=0
Nov 02 02:06:26.738794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.738756    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr" event={"ID":"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f","Type":"ContainerDied","Data":"c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b"}
Nov 02 02:06:26.738794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.738788    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr" event={"ID":"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f","Type":"ContainerDied","Data":"329375b87d5375c5f90fff27470f43c259cf64620bf87b948d697eb7c653aead"}
Nov 02 02:06:26.738859 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.738814    4912 scope.go:122] "RemoveContainer" containerID="c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b"
Nov 02 02:06:26.738931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.738918    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"
Nov 02 02:06:26.766171 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.766142    4912 scope.go:122] "RemoveContainer" containerID="c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b"
Nov 02 02:06:26.766556 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:06:26.766519    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b\": container with ID starting with c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b not found: ID does not exist" containerID="c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b"
Nov 02 02:06:26.766666 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.766559    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b"} err="failed to get container status \"c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b\": rpc error: code = NotFound desc = could not find container \"c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b\": container with ID starting with c03de5ee20a853360e21a1f6be1d7feeb93099f5d718d04bbf89eb771e73243b not found: ID does not exist"
Nov 02 02:06:26.793849 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.792722    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-kube-api-access-blj5r\" (UniqueName: \"kubernetes.io/projected/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-kube-api-access-blj5r\") pod \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\" (UID: \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\") "
Nov 02 02:06:26.793849 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.792795    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-plugins-config-volume\") pod \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\" (UID: \"62cd8a15-a27f-4c3f-aaa1-f8819bd6584f\") "
Nov 02 02:06:26.794553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.794245    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-plugins-config-volume" pod "62cd8a15-a27f-4c3f-aaa1-f8819bd6584f" (UID: "62cd8a15-a27f-4c3f-aaa1-f8819bd6584f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:26.797534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.797508    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-kube-api-access-blj5r" pod "62cd8a15-a27f-4c3f-aaa1-f8819bd6584f" (UID: "62cd8a15-a27f-4c3f-aaa1-f8819bd6584f"). InnerVolumeSpecName "kube-api-access-blj5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:26.839903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.839851    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:06:26.895624 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.895579    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blj5r\" (UniqueName: \"kubernetes.io/projected/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-kube-api-access-blj5r\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:26.895624 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.895618    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:26.946956 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.946926    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:26.997630 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.996670    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8167b26c-f67e-45fc-8691-95a15a9a6d92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8167b26c-f67e-45fc-8691-95a15a9a6d92-plugins-config-volume\") pod \"8167b26c-f67e-45fc-8691-95a15a9a6d92\" (UID: \"8167b26c-f67e-45fc-8691-95a15a9a6d92\") "
Nov 02 02:06:26.997630 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.996785    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8167b26c-f67e-45fc-8691-95a15a9a6d92-kube-api-access-5f9nd\" (UniqueName: \"kubernetes.io/projected/8167b26c-f67e-45fc-8691-95a15a9a6d92-kube-api-access-5f9nd\") pod \"8167b26c-f67e-45fc-8691-95a15a9a6d92\" (UID: \"8167b26c-f67e-45fc-8691-95a15a9a6d92\") "
Nov 02 02:06:26.998045 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:26.997993    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8167b26c-f67e-45fc-8691-95a15a9a6d92-plugins-config-volume" pod "8167b26c-f67e-45fc-8691-95a15a9a6d92" (UID: "8167b26c-f67e-45fc-8691-95a15a9a6d92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:27.000624 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.000594    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8167b26c-f67e-45fc-8691-95a15a9a6d92-kube-api-access-5f9nd" pod "8167b26c-f67e-45fc-8691-95a15a9a6d92" (UID: "8167b26c-f67e-45fc-8691-95a15a9a6d92"). InnerVolumeSpecName "kube-api-access-5f9nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:27.065755 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.065720    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"]
Nov 02 02:06:27.066894 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.066866    4912 cpu_manager.go:414] "containerMap: removing container" podUID="62cd8a15-a27f-4c3f-aaa1-f8819bd6584f" containerName="payload-processing"
Nov 02 02:06:27.066894 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.066895    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62cd8a15-a27f-4c3f-aaa1-f8819bd6584f" containerName="payload-processing"
Nov 02 02:06:27.067111 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.066926    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8167b26c-f67e-45fc-8691-95a15a9a6d92" containerName="payload-pre-processing"
Nov 02 02:06:27.067111 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.066934    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8167b26c-f67e-45fc-8691-95a15a9a6d92" containerName="payload-pre-processing"
Nov 02 02:06:27.067237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.067143    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="62cd8a15-a27f-4c3f-aaa1-f8819bd6584f" containerName="payload-processing"
Nov 02 02:06:27.067237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.067163    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8167b26c-f67e-45fc-8691-95a15a9a6d92" containerName="payload-pre-processing"
Nov 02 02:06:27.074427 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.072884    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.098630 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.098562    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8167b26c-f67e-45fc-8691-95a15a9a6d92-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:27.098630 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.098594    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9nd\" (UniqueName: \"kubernetes.io/projected/8167b26c-f67e-45fc-8691-95a15a9a6d92-kube-api-access-5f9nd\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:27.123856 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.123801    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"]
Nov 02 02:06:27.178951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.178916    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"]
Nov 02 02:06:27.186089 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.186037    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-129de4-5d6964c7b7-hx7cr"]
Nov 02 02:06:27.200425 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.200384    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rppz\" (UniqueName: \"kubernetes.io/projected/9c0d1913-f26e-43b9-a36f-04702cd3e560-kube-api-access-6rppz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.200592 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.200567    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.200637 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.200602    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.228703 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.228671    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62cd8a15-a27f-4c3f-aaa1-f8819bd6584f" path="/var/lib/kubelet/pods/62cd8a15-a27f-4c3f-aaa1-f8819bd6584f/volumes"
Nov 02 02:06:27.301944 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.301912    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rppz\" (UniqueName: \"kubernetes.io/projected/9c0d1913-f26e-43b9-a36f-04702cd3e560-kube-api-access-6rppz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.302162 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.302022    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.302162 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.302047    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.302713 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.302693    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.304767 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.304745    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.341336 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:06:27.341296    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf575054a_9c07_43be_83bd_867c03a71fe7.slice/crio-ae9b1b0052e1233240a0d46beebc7e7d8b2fa75a236b2d2647e905e766f5a267 WatchSource:0}: Error finding container ae9b1b0052e1233240a0d46beebc7e7d8b2fa75a236b2d2647e905e766f5a267: Status 404 returned error can't find the container with id ae9b1b0052e1233240a0d46beebc7e7d8b2fa75a236b2d2647e905e766f5a267
Nov 02 02:06:27.341733 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.341711    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"]
Nov 02 02:06:27.362848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.362786    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rppz\" (UniqueName: \"kubernetes.io/projected/9c0d1913-f26e-43b9-a36f-04702cd3e560-kube-api-access-6rppz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.394684 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.394658    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:27.749373 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.749337    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts" event={"ID":"f575054a-9c07-43be-83bd-867c03a71fe7","Type":"ContainerStarted","Data":"5599ff2565ff8c860ecaa20da7aaab84969470bd5559bf64a224c8d14dea2192"}
Nov 02 02:06:27.749548 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.749402    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts" event={"ID":"f575054a-9c07-43be-83bd-867c03a71fe7","Type":"ContainerStarted","Data":"ae9b1b0052e1233240a0d46beebc7e7d8b2fa75a236b2d2647e905e766f5a267"}
Nov 02 02:06:27.759148 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.759115    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r" event={"ID":"8167b26c-f67e-45fc-8691-95a15a9a6d92","Type":"ContainerDied","Data":"ea677f7f51dc83242be806c54f4a04a46189aebbc50b31910c7d4e80334b9edc"}
Nov 02 02:06:27.759295 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.759173    4912 scope.go:122] "RemoveContainer" containerID="96c3c6dbb8d37114a76793cb17a1734fd2dead00ce855a6010d1d16be69907c2"
Nov 02 02:06:27.759295 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.759119    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"
Nov 02 02:06:27.780093 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:06:27.780031    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0d1913_f26e_43b9_a36f_04702cd3e560.slice/crio-94b5627e881b0d7f964b380a4e8c0d13a14c4ff15064cc4e7b5057e4fa20d80f WatchSource:0}: Error finding container 94b5627e881b0d7f964b380a4e8c0d13a14c4ff15064cc4e7b5057e4fa20d80f: Status 404 returned error can't find the container with id 94b5627e881b0d7f964b380a4e8c0d13a14c4ff15064cc4e7b5057e4fa20d80f
Nov 02 02:06:27.782884 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.782855    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"]
Nov 02 02:06:27.865536 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:27.865472    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts" podStartSLOduration=1.865450331 podStartE2EDuration="1.865450331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:27.863467685 +0000 UTC m=+2080.768933041" watchObservedRunningTime="2026-11-02 02:06:27.865450331 +0000 UTC m=+2080.770915688"
Nov 02 02:06:28.020609 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:28.018462    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"]
Nov 02 02:06:28.025081 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:28.024923    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-129de4-68df866ccf-mfj6r"]
Nov 02 02:06:28.773807 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:28.773705    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k" event={"ID":"9c0d1913-f26e-43b9-a36f-04702cd3e560","Type":"ContainerStarted","Data":"c5a9218ae4d1c4b44b78d82c5522892bedd0af2ee09dd3c1a018b7e9adf88f52"}
Nov 02 02:06:28.773807 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:28.773771    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k" event={"ID":"9c0d1913-f26e-43b9-a36f-04702cd3e560","Type":"ContainerStarted","Data":"94b5627e881b0d7f964b380a4e8c0d13a14c4ff15064cc4e7b5057e4fa20d80f"}
Nov 02 02:06:28.776108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:28.776043    4912 generic.go:356] "Generic (PLEG): container finished" podID="f575054a-9c07-43be-83bd-867c03a71fe7" containerID="5599ff2565ff8c860ecaa20da7aaab84969470bd5559bf64a224c8d14dea2192" exitCode=0
Nov 02 02:06:28.776244 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:28.776135    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts" event={"ID":"f575054a-9c07-43be-83bd-867c03a71fe7","Type":"ContainerDied","Data":"5599ff2565ff8c860ecaa20da7aaab84969470bd5559bf64a224c8d14dea2192"}
Nov 02 02:06:28.902107 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:28.900238    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k" podStartSLOduration=1.9002166950000001 podStartE2EDuration="1.900216695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:28.897392143 +0000 UTC m=+2081.802857500" watchObservedRunningTime="2026-11-02 02:06:28.900216695 +0000 UTC m=+2081.805682052"
Nov 02 02:06:28.902107 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:28.900824    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:29.227217 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:29.227184    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8167b26c-f67e-45fc-8691-95a15a9a6d92" path="/var/lib/kubelet/pods/8167b26c-f67e-45fc-8691-95a15a9a6d92/volumes"
Nov 02 02:06:29.788943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:29.788907    4912 generic.go:356] "Generic (PLEG): container finished" podID="9c0d1913-f26e-43b9-a36f-04702cd3e560" containerID="c5a9218ae4d1c4b44b78d82c5522892bedd0af2ee09dd3c1a018b7e9adf88f52" exitCode=0
Nov 02 02:06:29.789383 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:29.788995    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k" event={"ID":"9c0d1913-f26e-43b9-a36f-04702cd3e560","Type":"ContainerDied","Data":"c5a9218ae4d1c4b44b78d82c5522892bedd0af2ee09dd3c1a018b7e9adf88f52"}
Nov 02 02:06:30.004830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.004802    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:30.152635 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.152601    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f575054a-9c07-43be-83bd-867c03a71fe7-kube-api-access-cmb4c\" (UniqueName: \"kubernetes.io/projected/f575054a-9c07-43be-83bd-867c03a71fe7-kube-api-access-cmb4c\") pod \"f575054a-9c07-43be-83bd-867c03a71fe7\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") "
Nov 02 02:06:30.152833 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.152654    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-xks-ca\") pod \"f575054a-9c07-43be-83bd-867c03a71fe7\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") "
Nov 02 02:06:30.152902 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.152872    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-openshift-ca\") pod \"f575054a-9c07-43be-83bd-867c03a71fe7\" (UID: \"f575054a-9c07-43be-83bd-867c03a71fe7\") "
Nov 02 02:06:30.154650 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.154609    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-openshift-ca" pod "f575054a-9c07-43be-83bd-867c03a71fe7" (UID: "f575054a-9c07-43be-83bd-867c03a71fe7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:30.155808 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.155777    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-xks-ca" pod "f575054a-9c07-43be-83bd-867c03a71fe7" (UID: "f575054a-9c07-43be-83bd-867c03a71fe7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:30.155900 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.155867    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f575054a-9c07-43be-83bd-867c03a71fe7-kube-api-access-cmb4c" pod "f575054a-9c07-43be-83bd-867c03a71fe7" (UID: "f575054a-9c07-43be-83bd-867c03a71fe7"). InnerVolumeSpecName "kube-api-access-cmb4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:30.255772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.255737    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmb4c\" (UniqueName: \"kubernetes.io/projected/f575054a-9c07-43be-83bd-867c03a71fe7-kube-api-access-cmb4c\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:30.255772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.255772    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:30.256035 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.255788    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f575054a-9c07-43be-83bd-867c03a71fe7-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:30.799596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.799563    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"
Nov 02 02:06:30.799596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.799581    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts" event={"ID":"f575054a-9c07-43be-83bd-867c03a71fe7","Type":"ContainerDied","Data":"ae9b1b0052e1233240a0d46beebc7e7d8b2fa75a236b2d2647e905e766f5a267"}
Nov 02 02:06:30.800154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.799633    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae9b1b0052e1233240a0d46beebc7e7d8b2fa75a236b2d2647e905e766f5a267"
Nov 02 02:06:30.983283 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:30.983255    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:31.169376 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.169341    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-openshift-ca\") pod \"9c0d1913-f26e-43b9-a36f-04702cd3e560\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") "
Nov 02 02:06:31.169557 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.169395    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0d1913-f26e-43b9-a36f-04702cd3e560-kube-api-access-6rppz\" (UniqueName: \"kubernetes.io/projected/9c0d1913-f26e-43b9-a36f-04702cd3e560-kube-api-access-6rppz\") pod \"9c0d1913-f26e-43b9-a36f-04702cd3e560\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") "
Nov 02 02:06:31.169649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.169622    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-xks-ca\") pod \"9c0d1913-f26e-43b9-a36f-04702cd3e560\" (UID: \"9c0d1913-f26e-43b9-a36f-04702cd3e560\") "
Nov 02 02:06:31.170544 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.170102    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-openshift-ca" pod "9c0d1913-f26e-43b9-a36f-04702cd3e560" (UID: "9c0d1913-f26e-43b9-a36f-04702cd3e560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:31.170544 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.170258    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:31.172289 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.172097    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-xks-ca" pod "9c0d1913-f26e-43b9-a36f-04702cd3e560" (UID: "9c0d1913-f26e-43b9-a36f-04702cd3e560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:31.172673 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.172646    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0d1913-f26e-43b9-a36f-04702cd3e560-kube-api-access-6rppz" pod "9c0d1913-f26e-43b9-a36f-04702cd3e560" (UID: "9c0d1913-f26e-43b9-a36f-04702cd3e560"). InnerVolumeSpecName "kube-api-access-6rppz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:31.271797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.271767    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rppz\" (UniqueName: \"kubernetes.io/projected/9c0d1913-f26e-43b9-a36f-04702cd3e560-kube-api-access-6rppz\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:31.271797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.271797    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0d1913-f26e-43b9-a36f-04702cd3e560-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:31.809224 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.809190    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k" event={"ID":"9c0d1913-f26e-43b9-a36f-04702cd3e560","Type":"ContainerDied","Data":"94b5627e881b0d7f964b380a4e8c0d13a14c4ff15064cc4e7b5057e4fa20d80f"}
Nov 02 02:06:31.809735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.809237    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94b5627e881b0d7f964b380a4e8c0d13a14c4ff15064cc4e7b5057e4fa20d80f"
Nov 02 02:06:31.809735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:31.809245    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"
Nov 02 02:06:34.252119 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:34.252040    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:38.194985 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:38.194947    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:38.427830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:38.427786    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:39.578220 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:39.578178    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:40.037686 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:40.037651    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:40.214365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:40.214327    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:40.401109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:40.401075    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:40.750747 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:40.750657    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"]
Nov 02 02:06:40.800390 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:40.800358    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd3971-555fz"]
Nov 02 02:06:41.228390 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:41.228356    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb" path="/var/lib/kubelet/pods/82eb2b69-4fa5-4dee-a4c5-c7e0e47ad0fb/volumes"
Nov 02 02:06:41.265013 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:41.264983    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:41.577130 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:41.577020    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:42.141037 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:06:42.140993    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:06:44.348365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:44.348331    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:44.619140 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:44.619086    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:44.836893 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:44.836860    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"]
Nov 02 02:06:44.884094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:44.883991    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a24b9-qmvts"]
Nov 02 02:06:45.227806 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:45.227729    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f575054a-9c07-43be-83bd-867c03a71fe7" path="/var/lib/kubelet/pods/f575054a-9c07-43be-83bd-867c03a71fe7/volumes"
Nov 02 02:06:45.482856 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:45.482766    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:46.724752 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:46.724697    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:47.633123 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:47.633044    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:47.729121 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:47.729095    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:06:47.736667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:47.736644    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:06:47.808473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:47.808441    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:47.857071 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:47.857033    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:06:47.863992 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:47.863971    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:06:48.072915 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:48.072824    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:49.803267 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:49.803226    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:51.160863 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.160829    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:51.366527 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.366490    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"]
Nov 02 02:06:51.367340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.367324    4912 cpu_manager.go:414] "containerMap: removing container" podUID="9c0d1913-f26e-43b9-a36f-04702cd3e560" containerName="revoke-keys"
Nov 02 02:06:51.367400 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.367341    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0d1913-f26e-43b9-a36f-04702cd3e560" containerName="revoke-keys"
Nov 02 02:06:51.367400 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.367355    4912 cpu_manager.go:414] "containerMap: removing container" podUID="f575054a-9c07-43be-83bd-867c03a71fe7" containerName="revoke-keys"
Nov 02 02:06:51.367400 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.367361    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f575054a-9c07-43be-83bd-867c03a71fe7" containerName="revoke-keys"
Nov 02 02:06:51.367512 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.367476    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0d1913-f26e-43b9-a36f-04702cd3e560" containerName="revoke-keys"
Nov 02 02:06:51.367512 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.367485    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="f575054a-9c07-43be-83bd-867c03a71fe7" containerName="revoke-keys"
Nov 02 02:06:51.372736 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.372717    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.425133 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.425017    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:06:51.425318 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.425257    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:06:51.427573 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.427549    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"]
Nov 02 02:06:51.495709 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.495670    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22b5d4d-cbm25\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.495896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.495800    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98xt\" (UniqueName: \"kubernetes.io/projected/6c922f73-2e92-4ea2-895b-f0b9ac10add4-kube-api-access-w98xt\") pod \"maas-api-revoke-sub-f22b5d4d-cbm25\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.495896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.495839    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22b5d4d-cbm25\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.597112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.597077    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22b5d4d-cbm25\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.597331 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.597175    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98xt\" (UniqueName: \"kubernetes.io/projected/6c922f73-2e92-4ea2-895b-f0b9ac10add4-kube-api-access-w98xt\") pod \"maas-api-revoke-sub-f22b5d4d-cbm25\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.597331 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.597210    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22b5d4d-cbm25\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.597948 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.597920    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22b5d4d-cbm25\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.599730 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.599707    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22b5d4d-cbm25\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.661784 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.661754    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98xt\" (UniqueName: \"kubernetes.io/projected/6c922f73-2e92-4ea2-895b-f0b9ac10add4-kube-api-access-w98xt\") pod \"maas-api-revoke-sub-f22b5d4d-cbm25\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.688856 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.688773    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:51.870074 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:51.870022    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:52.057424 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:52.057337    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:52.239644 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:52.239606    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:52.259702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:52.259636    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"]
Nov 02 02:06:52.263186 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:06:52.262992    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c922f73_2e92_4ea2_895b_f0b9ac10add4.slice/crio-877a2c8f8d01326859c21aa744b21d593f7f88ac915f8e8273d715e39ca82d57 WatchSource:0}: Error finding container 877a2c8f8d01326859c21aa744b21d593f7f88ac915f8e8273d715e39ca82d57: Status 404 returned error can't find the container with id 877a2c8f8d01326859c21aa744b21d593f7f88ac915f8e8273d715e39ca82d57
Nov 02 02:06:52.431673 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:52.431641    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:53.063845 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:53.063810    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25" event={"ID":"6c922f73-2e92-4ea2-895b-f0b9ac10add4","Type":"ContainerStarted","Data":"ffe5e21773bde94ba3cf3567f0298ff6e58110430ecfd7e95fd57f9ac57df425"}
Nov 02 02:06:53.064044 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:53.063884    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25" event={"ID":"6c922f73-2e92-4ea2-895b-f0b9ac10add4","Type":"ContainerStarted","Data":"877a2c8f8d01326859c21aa744b21d593f7f88ac915f8e8273d715e39ca82d57"}
Nov 02 02:06:53.182252 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:53.182196    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25" podStartSLOduration=2.182182355 podStartE2EDuration="2.182182355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:53.177873501 +0000 UTC m=+2106.083338858" watchObservedRunningTime="2026-11-02 02:06:53.182182355 +0000 UTC m=+2106.087647706"
Nov 02 02:06:54.077964 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:54.077928    4912 generic.go:356] "Generic (PLEG): container finished" podID="6c922f73-2e92-4ea2-895b-f0b9ac10add4" containerID="ffe5e21773bde94ba3cf3567f0298ff6e58110430ecfd7e95fd57f9ac57df425" exitCode=0
Nov 02 02:06:54.078425 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:54.078027    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25" event={"ID":"6c922f73-2e92-4ea2-895b-f0b9ac10add4","Type":"ContainerDied","Data":"ffe5e21773bde94ba3cf3567f0298ff6e58110430ecfd7e95fd57f9ac57df425"}
Nov 02 02:06:54.974733 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:54.974699    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"]
Nov 02 02:06:54.979683 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:54.979663    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.021915 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.021869    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa1597-886kx\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.022185 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.022160    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7qd\" (UniqueName: \"kubernetes.io/projected/63d50c81-35f7-4504-924c-6419fac220c0-kube-api-access-2j7qd\") pod \"maas-api-revoke-keys-e2e-inf-a-aa1597-886kx\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.022261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.022197    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa1597-886kx\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.036788 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.036752    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"]
Nov 02 02:06:55.123739 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.123706    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7qd\" (UniqueName: \"kubernetes.io/projected/63d50c81-35f7-4504-924c-6419fac220c0-kube-api-access-2j7qd\") pod \"maas-api-revoke-keys-e2e-inf-a-aa1597-886kx\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.123739 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.123744    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa1597-886kx\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.124300 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.123868    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa1597-886kx\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.124804 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.124755    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa1597-886kx\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.126932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.126910    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa1597-886kx\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.188281 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.187174    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7qd\" (UniqueName: \"kubernetes.io/projected/63d50c81-35f7-4504-924c-6419fac220c0-kube-api-access-2j7qd\") pod \"maas-api-revoke-keys-e2e-inf-a-aa1597-886kx\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.270839 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.270814    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:55.299911 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.299870    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:55.326299 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.326271    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-openshift-ca\") pod \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") "
Nov 02 02:06:55.326442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.326357    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c922f73-2e92-4ea2-895b-f0b9ac10add4-kube-api-access-w98xt\" (UniqueName: \"kubernetes.io/projected/6c922f73-2e92-4ea2-895b-f0b9ac10add4-kube-api-access-w98xt\") pod \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") "
Nov 02 02:06:55.326442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.326396    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-xks-ca\") pod \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\" (UID: \"6c922f73-2e92-4ea2-895b-f0b9ac10add4\") "
Nov 02 02:06:55.326669 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.326644    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-openshift-ca" pod "6c922f73-2e92-4ea2-895b-f0b9ac10add4" (UID: "6c922f73-2e92-4ea2-895b-f0b9ac10add4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:55.327365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.327332    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:55.328781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.328756    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-xks-ca" pod "6c922f73-2e92-4ea2-895b-f0b9ac10add4" (UID: "6c922f73-2e92-4ea2-895b-f0b9ac10add4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:55.328862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.328778    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c922f73-2e92-4ea2-895b-f0b9ac10add4-kube-api-access-w98xt" pod "6c922f73-2e92-4ea2-895b-f0b9ac10add4" (UID: "6c922f73-2e92-4ea2-895b-f0b9ac10add4"). InnerVolumeSpecName "kube-api-access-w98xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:55.354363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.354330    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"]
Nov 02 02:06:55.355078 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.355046    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6c922f73-2e92-4ea2-895b-f0b9ac10add4" containerName="revoke-keys"
Nov 02 02:06:55.355137 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.355081    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c922f73-2e92-4ea2-895b-f0b9ac10add4" containerName="revoke-keys"
Nov 02 02:06:55.355226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.355215    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c922f73-2e92-4ea2-895b-f0b9ac10add4" containerName="revoke-keys"
Nov 02 02:06:55.359995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.359968    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.413664 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.413618    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"]
Nov 02 02:06:55.428581 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.428548    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.428815 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.428588    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97w4\" (UniqueName: \"kubernetes.io/projected/69f90746-00e6-45fa-89a7-c7a5ade39f7f-kube-api-access-x97w4\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.428815 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.428652    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.429126 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.429101    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w98xt\" (UniqueName: \"kubernetes.io/projected/6c922f73-2e92-4ea2-895b-f0b9ac10add4-kube-api-access-w98xt\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:55.429177 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.429135    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c922f73-2e92-4ea2-895b-f0b9ac10add4-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:55.530434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.530345    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.530434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.530382    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97w4\" (UniqueName: \"kubernetes.io/projected/69f90746-00e6-45fa-89a7-c7a5ade39f7f-kube-api-access-x97w4\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.530434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.530424    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.531170 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.531147    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.532969 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.532940    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.591496 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.591460    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97w4\" (UniqueName: \"kubernetes.io/projected/69f90746-00e6-45fa-89a7-c7a5ade39f7f-kube-api-access-x97w4\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.686967 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.686937    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:55.689175 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:06:55.689142    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d50c81_35f7_4504_924c_6419fac220c0.slice/crio-af4847a741adb60ba2f0e55b2c87f3a820dc9b4229cb5289ca7c186a4b00ec58 WatchSource:0}: Error finding container af4847a741adb60ba2f0e55b2c87f3a820dc9b4229cb5289ca7c186a4b00ec58: Status 404 returned error can't find the container with id af4847a741adb60ba2f0e55b2c87f3a820dc9b4229cb5289ca7c186a4b00ec58
Nov 02 02:06:55.691202 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:55.691173    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"]
Nov 02 02:06:56.076097 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:06:56.076037    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f90746_00e6_45fa_89a7_c7a5ade39f7f.slice/crio-53b06ef1b0edff9e18d80597c4174c54f3afecd6df555760e72501858efd1d4a WatchSource:0}: Error finding container 53b06ef1b0edff9e18d80597c4174c54f3afecd6df555760e72501858efd1d4a: Status 404 returned error can't find the container with id 53b06ef1b0edff9e18d80597c4174c54f3afecd6df555760e72501858efd1d4a
Nov 02 02:06:56.076784 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:56.076759    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"]
Nov 02 02:06:56.100196 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:56.100161    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx" event={"ID":"63d50c81-35f7-4504-924c-6419fac220c0","Type":"ContainerStarted","Data":"cc4537bd75304114625ac880c2bfa08ee3280f7d72f8e66132cbb22e9b698795"}
Nov 02 02:06:56.100312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:56.100227    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx" event={"ID":"63d50c81-35f7-4504-924c-6419fac220c0","Type":"ContainerStarted","Data":"af4847a741adb60ba2f0e55b2c87f3a820dc9b4229cb5289ca7c186a4b00ec58"}
Nov 02 02:06:56.102166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:56.102146    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"
Nov 02 02:06:56.102413 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:56.102183    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25" event={"ID":"6c922f73-2e92-4ea2-895b-f0b9ac10add4","Type":"ContainerDied","Data":"877a2c8f8d01326859c21aa744b21d593f7f88ac915f8e8273d715e39ca82d57"}
Nov 02 02:06:56.102413 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:56.102236    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="877a2c8f8d01326859c21aa744b21d593f7f88ac915f8e8273d715e39ca82d57"
Nov 02 02:06:56.103761 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:56.103740    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d" event={"ID":"69f90746-00e6-45fa-89a7-c7a5ade39f7f","Type":"ContainerStarted","Data":"53b06ef1b0edff9e18d80597c4174c54f3afecd6df555760e72501858efd1d4a"}
Nov 02 02:06:56.216519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:56.216422    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx" podStartSLOduration=2.216407177 podStartE2EDuration="2.216407177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:56.21521348 +0000 UTC m=+2109.120678838" watchObservedRunningTime="2026-11-02 02:06:56.216407177 +0000 UTC m=+2109.121872535"
Nov 02 02:06:57.115494 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:57.115456    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d" event={"ID":"69f90746-00e6-45fa-89a7-c7a5ade39f7f","Type":"ContainerStarted","Data":"f47bcba3686e96789e679662c630bc0781debaa53410f83dd48b6d3ec28e6e74"}
Nov 02 02:06:57.117521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:57.117489    4912 generic.go:356] "Generic (PLEG): container finished" podID="63d50c81-35f7-4504-924c-6419fac220c0" containerID="cc4537bd75304114625ac880c2bfa08ee3280f7d72f8e66132cbb22e9b698795" exitCode=0
Nov 02 02:06:57.117682 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:57.117569    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx" event={"ID":"63d50c81-35f7-4504-924c-6419fac220c0","Type":"ContainerDied","Data":"cc4537bd75304114625ac880c2bfa08ee3280f7d72f8e66132cbb22e9b698795"}
Nov 02 02:06:57.240104 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:57.240002    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d" podStartSLOduration=2.239983657 podStartE2EDuration="2.239983657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:57.230988401 +0000 UTC m=+2110.136453758" watchObservedRunningTime="2026-11-02 02:06:57.239983657 +0000 UTC m=+2110.145449015"
Nov 02 02:06:57.329833 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:57.329794    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:06:58.127592 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.127557    4912 generic.go:356] "Generic (PLEG): container finished" podID="69f90746-00e6-45fa-89a7-c7a5ade39f7f" containerID="f47bcba3686e96789e679662c630bc0781debaa53410f83dd48b6d3ec28e6e74" exitCode=0
Nov 02 02:06:58.127778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.127617    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d" event={"ID":"69f90746-00e6-45fa-89a7-c7a5ade39f7f","Type":"ContainerDied","Data":"f47bcba3686e96789e679662c630bc0781debaa53410f83dd48b6d3ec28e6e74"}
Nov 02 02:06:58.335966 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.335939    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:58.377137 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.377111    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-openshift-ca\") pod \"63d50c81-35f7-4504-924c-6419fac220c0\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") "
Nov 02 02:06:58.377361 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.377214    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d50c81-35f7-4504-924c-6419fac220c0-kube-api-access-2j7qd\" (UniqueName: \"kubernetes.io/projected/63d50c81-35f7-4504-924c-6419fac220c0-kube-api-access-2j7qd\") pod \"63d50c81-35f7-4504-924c-6419fac220c0\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") "
Nov 02 02:06:58.377361 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.377253    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-xks-ca\") pod \"63d50c81-35f7-4504-924c-6419fac220c0\" (UID: \"63d50c81-35f7-4504-924c-6419fac220c0\") "
Nov 02 02:06:58.377574 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.377531    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-openshift-ca" pod "63d50c81-35f7-4504-924c-6419fac220c0" (UID: "63d50c81-35f7-4504-924c-6419fac220c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:58.377890 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.377828    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:58.379539 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.379517    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-xks-ca" pod "63d50c81-35f7-4504-924c-6419fac220c0" (UID: "63d50c81-35f7-4504-924c-6419fac220c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:58.379641 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.379599    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d50c81-35f7-4504-924c-6419fac220c0-kube-api-access-2j7qd" pod "63d50c81-35f7-4504-924c-6419fac220c0" (UID: "63d50c81-35f7-4504-924c-6419fac220c0"). InnerVolumeSpecName "kube-api-access-2j7qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:58.478994 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.478959    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j7qd\" (UniqueName: \"kubernetes.io/projected/63d50c81-35f7-4504-924c-6419fac220c0-kube-api-access-2j7qd\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:58.478994 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:58.478995    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d50c81-35f7-4504-924c-6419fac220c0-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:59.136534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.136509    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"
Nov 02 02:06:59.136726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.136548    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx" event={"ID":"63d50c81-35f7-4504-924c-6419fac220c0","Type":"ContainerDied","Data":"af4847a741adb60ba2f0e55b2c87f3a820dc9b4229cb5289ca7c186a4b00ec58"}
Nov 02 02:06:59.136726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.136597    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4847a741adb60ba2f0e55b2c87f3a820dc9b4229cb5289ca7c186a4b00ec58"
Nov 02 02:06:59.333710 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.333676    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:06:59.394620 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.394532    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f90746-00e6-45fa-89a7-c7a5ade39f7f-kube-api-access-x97w4\" (UniqueName: \"kubernetes.io/projected/69f90746-00e6-45fa-89a7-c7a5ade39f7f-kube-api-access-x97w4\") pod \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") "
Nov 02 02:06:59.395128 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.394698    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-xks-ca\") pod \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") "
Nov 02 02:06:59.395128 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.394798    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-openshift-ca\") pod \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\" (UID: \"69f90746-00e6-45fa-89a7-c7a5ade39f7f\") "
Nov 02 02:06:59.395269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.395219    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-openshift-ca" pod "69f90746-00e6-45fa-89a7-c7a5ade39f7f" (UID: "69f90746-00e6-45fa-89a7-c7a5ade39f7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:59.395483 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.395460    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:59.397375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.397347    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f90746-00e6-45fa-89a7-c7a5ade39f7f-kube-api-access-x97w4" pod "69f90746-00e6-45fa-89a7-c7a5ade39f7f" (UID: "69f90746-00e6-45fa-89a7-c7a5ade39f7f"). InnerVolumeSpecName "kube-api-access-x97w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:59.397507 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.397450    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-xks-ca" pod "69f90746-00e6-45fa-89a7-c7a5ade39f7f" (UID: "69f90746-00e6-45fa-89a7-c7a5ade39f7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:59.498160 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.497994    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x97w4\" (UniqueName: \"kubernetes.io/projected/69f90746-00e6-45fa-89a7-c7a5ade39f7f-kube-api-access-x97w4\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:59.498160 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.498031    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f90746-00e6-45fa-89a7-c7a5ade39f7f-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:59.589991 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:06:59.589951    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:00.146109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:00.146069    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d" event={"ID":"69f90746-00e6-45fa-89a7-c7a5ade39f7f","Type":"ContainerDied","Data":"53b06ef1b0edff9e18d80597c4174c54f3afecd6df555760e72501858efd1d4a"}
Nov 02 02:07:00.146109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:00.146108    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"
Nov 02 02:07:00.146331 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:00.146127    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53b06ef1b0edff9e18d80597c4174c54f3afecd6df555760e72501858efd1d4a"
Nov 02 02:07:00.545533 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:00.545443    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:00.764491 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:00.764456    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:01.116085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.107509    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"]
Nov 02 02:07:01.116085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.108603    4912 cpu_manager.go:414] "containerMap: removing container" podUID="63d50c81-35f7-4504-924c-6419fac220c0" containerName="revoke-keys"
Nov 02 02:07:01.116085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.108626    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d50c81-35f7-4504-924c-6419fac220c0" containerName="revoke-keys"
Nov 02 02:07:01.116085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.108690    4912 cpu_manager.go:414] "containerMap: removing container" podUID="69f90746-00e6-45fa-89a7-c7a5ade39f7f" containerName="revoke-keys"
Nov 02 02:07:01.116085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.108699    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f90746-00e6-45fa-89a7-c7a5ade39f7f" containerName="revoke-keys"
Nov 02 02:07:01.116085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.108888    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f90746-00e6-45fa-89a7-c7a5ade39f7f" containerName="revoke-keys"
Nov 02 02:07:01.116085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.108911    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d50c81-35f7-4504-924c-6419fac220c0" containerName="revoke-keys"
Nov 02 02:07:01.131540 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.131511    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:01.169498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.169460    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"]
Nov 02 02:07:01.232359 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.232293    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq\" (UID: \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:01.232683 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.232649    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zspkw\" (UniqueName: \"kubernetes.io/projected/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-kube-api-access-zspkw\") pod \"payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq\" (UID: \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:01.270030 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.270003    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1c942d-dockercfg-2gxhj\""
Nov 02 02:07:01.270277 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.270083    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1c942d\""
Nov 02 02:07:01.278007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.277978    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"]
Nov 02 02:07:01.283959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.283937    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:01.316508 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.316480    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"]
Nov 02 02:07:01.334271 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.334234    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbprf\" (UniqueName: \"kubernetes.io/projected/717f0211-196d-4bc5-8725-a9a6d8bae6d1-kube-api-access-kbprf\") pod \"payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n\" (UID: \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:01.334548 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.334520    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f0211-196d-4bc5-8725-a9a6d8bae6d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n\" (UID: \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:01.334778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.334756    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq\" (UID: \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:01.334863 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.334838    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zspkw\" (UniqueName: \"kubernetes.io/projected/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-kube-api-access-zspkw\") pod \"payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq\" (UID: \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:01.335914 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.335889    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq\" (UID: \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:01.396148 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.396027    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zspkw\" (UniqueName: \"kubernetes.io/projected/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-kube-api-access-zspkw\") pod \"payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq\" (UID: \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:01.436166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.436132    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f0211-196d-4bc5-8725-a9a6d8bae6d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n\" (UID: \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:01.436384 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.436369    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbprf\" (UniqueName: \"kubernetes.io/projected/717f0211-196d-4bc5-8725-a9a6d8bae6d1-kube-api-access-kbprf\") pod \"payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n\" (UID: \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:01.436899 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.436880    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f0211-196d-4bc5-8725-a9a6d8bae6d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n\" (UID: \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:01.497742 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.497705    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbprf\" (UniqueName: \"kubernetes.io/projected/717f0211-196d-4bc5-8725-a9a6d8bae6d1-kube-api-access-kbprf\") pod \"payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n\" (UID: \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:01.549328 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.549297    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:01.601641 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.601604    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:01.936403 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:07:01.936361    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafd3ce7_9463_4588_9fa5_2c9e43f6f5b6.slice/crio-b17ae4326c26d685503b86834acd90db459bd763110a07b165fc2c583f97cc6c WatchSource:0}: Error finding container b17ae4326c26d685503b86834acd90db459bd763110a07b165fc2c583f97cc6c: Status 404 returned error can't find the container with id b17ae4326c26d685503b86834acd90db459bd763110a07b165fc2c583f97cc6c
Nov 02 02:07:01.936943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:01.936919    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"]
Nov 02 02:07:02.048253 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:07:02.048223    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717f0211_196d_4bc5_8725_a9a6d8bae6d1.slice/crio-a2a8935eafb32629f62453c99519f08d7f45b05def14f5680c86c3bd067bfbcf WatchSource:0}: Error finding container a2a8935eafb32629f62453c99519f08d7f45b05def14f5680c86c3bd067bfbcf: Status 404 returned error can't find the container with id a2a8935eafb32629f62453c99519f08d7f45b05def14f5680c86c3bd067bfbcf
Nov 02 02:07:02.048401 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.048285    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"]
Nov 02 02:07:02.239442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.239344    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n" event={"ID":"717f0211-196d-4bc5-8725-a9a6d8bae6d1","Type":"ContainerStarted","Data":"261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2"}
Nov 02 02:07:02.239442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.239413    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:02.239442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.239438    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n" event={"ID":"717f0211-196d-4bc5-8725-a9a6d8bae6d1","Type":"ContainerStarted","Data":"a2a8935eafb32629f62453c99519f08d7f45b05def14f5680c86c3bd067bfbcf"}
Nov 02 02:07:02.241216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.241189    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq" event={"ID":"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6","Type":"ContainerStarted","Data":"2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20"}
Nov 02 02:07:02.241341 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.241232    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq" event={"ID":"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6","Type":"ContainerStarted","Data":"b17ae4326c26d685503b86834acd90db459bd763110a07b165fc2c583f97cc6c"}
Nov 02 02:07:02.241341 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.241269    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:02.359700 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.359637    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n" podStartSLOduration=1.359607855 podStartE2EDuration="1.359607855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:02.354195431 +0000 UTC m=+2115.259660787" watchObservedRunningTime="2026-11-02 02:07:02.359607855 +0000 UTC m=+2115.265073214"
Nov 02 02:07:02.486083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.484635    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq" podStartSLOduration=1.484612218 podStartE2EDuration="1.484612218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:02.46902746 +0000 UTC m=+2115.374492818" watchObservedRunningTime="2026-11-02 02:07:02.484612218 +0000 UTC m=+2115.390077577"
Nov 02 02:07:02.657681 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:02.657642    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:04.830093 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:04.829995    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:04.999827 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:04.999790    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:06.547955 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:06.547917    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"]
Nov 02 02:07:06.797878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:06.797844    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"]
Nov 02 02:07:06.798114 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:06.798046    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr" podUID="b7d15873-c164-4c41-aac9-bff2fa0f72fe" containerName="maas-api" containerID="cri-o://0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39" gracePeriod=30
Nov 02 02:07:06.922819 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:06.922789    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:07.102951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.102918    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:07.130141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.130116    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:07:07.231467 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.231431    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls\") pod \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") "
Nov 02 02:07:07.231797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.231776    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d15873-c164-4c41-aac9-bff2fa0f72fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d15873-c164-4c41-aac9-bff2fa0f72fe-openshift-service-ca\") pod \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") "
Nov 02 02:07:07.232154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.232122    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d15873-c164-4c41-aac9-bff2fa0f72fe-openshift-service-ca" pod "b7d15873-c164-4c41-aac9-bff2fa0f72fe" (UID: "b7d15873-c164-4c41-aac9-bff2fa0f72fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:07.232285 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.232267    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d15873-c164-4c41-aac9-bff2fa0f72fe-kube-api-access-ccpl9\" (UniqueName: \"kubernetes.io/projected/b7d15873-c164-4c41-aac9-bff2fa0f72fe-kube-api-access-ccpl9\") pod \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") "
Nov 02 02:07:07.232434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.232417    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-metrics-certs\") pod \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\" (UID: \"b7d15873-c164-4c41-aac9-bff2fa0f72fe\") "
Nov 02 02:07:07.233113 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.233081    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d15873-c164-4c41-aac9-bff2fa0f72fe-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:07.295876 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.295692    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d15873-c164-4c41-aac9-bff2fa0f72fe-kube-api-access-ccpl9" pod "b7d15873-c164-4c41-aac9-bff2fa0f72fe" (UID: "b7d15873-c164-4c41-aac9-bff2fa0f72fe"). InnerVolumeSpecName "kube-api-access-ccpl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:07.295876 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.295840    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls" pod "b7d15873-c164-4c41-aac9-bff2fa0f72fe" (UID: "b7d15873-c164-4c41-aac9-bff2fa0f72fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:07.299099 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.299018    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-metrics-certs" pod "b7d15873-c164-4c41-aac9-bff2fa0f72fe" (UID: "b7d15873-c164-4c41-aac9-bff2fa0f72fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:07.313806 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.313777    4912 generic.go:356] "Generic (PLEG): container finished" podID="b7d15873-c164-4c41-aac9-bff2fa0f72fe" containerID="0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39" exitCode=0
Nov 02 02:07:07.313988 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.313839    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"
Nov 02 02:07:07.313988 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.313856    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr" event={"ID":"b7d15873-c164-4c41-aac9-bff2fa0f72fe","Type":"ContainerDied","Data":"0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39"}
Nov 02 02:07:07.313988 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.313915    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr" event={"ID":"b7d15873-c164-4c41-aac9-bff2fa0f72fe","Type":"ContainerDied","Data":"c215a5900ee3b7ad75a743893ad31d342ca1482d4235f52e9da4fc922fc8d3f3"}
Nov 02 02:07:07.313988 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.313942    4912 scope.go:122] "RemoveContainer" containerID="0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39"
Nov 02 02:07:07.326591 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.326571    4912 scope.go:122] "RemoveContainer" containerID="0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39"
Nov 02 02:07:07.326907 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:07:07.326888    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39\": container with ID starting with 0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39 not found: ID does not exist" containerID="0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39"
Nov 02 02:07:07.326952 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.326917    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39"} err="failed to get container status \"0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39\": rpc error: code = NotFound desc = could not find container \"0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39\": container with ID starting with 0fb6d900a9ded3d24322f3e8c921722ec2db84050eda52d5b94b116e88fc3e39 not found: ID does not exist"
Nov 02 02:07:07.334368 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.334336    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccpl9\" (UniqueName: \"kubernetes.io/projected/b7d15873-c164-4c41-aac9-bff2fa0f72fe-kube-api-access-ccpl9\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:07.334490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.334372    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:07.334490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.334389    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d15873-c164-4c41-aac9-bff2fa0f72fe-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:07.344371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.344341    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"]
Nov 02 02:07:07.475668 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.475626    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:07.579188 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.579154    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"]
Nov 02 02:07:07.583710 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.583682    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa1597-675fbc7dc6-sj2sr"]
Nov 02 02:07:07.727911 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:07.727824    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:08.861191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:08.860899    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:09.227284 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:09.227250    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d15873-c164-4c41-aac9-bff2fa0f72fe" path="/var/lib/kubelet/pods/b7d15873-c164-4c41-aac9-bff2fa0f72fe/volumes"
Nov 02 02:07:10.279155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:10.279101    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:10.449605 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:10.449559    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:11.248290 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.248258    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:11.481030 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.480910    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:11.549607 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.549514    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns" podUID="b05adf78-f5e8-45dd-87a0-fd8c04cb1c79" containerName="payload-pre-processing" containerID="cri-o://201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0" gracePeriod=25
Nov 02 02:07:11.703808 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.703771    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"]
Nov 02 02:07:11.752391 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.752360    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa1597-886kx"]
Nov 02 02:07:11.838757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.838726    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 02:07:11.943843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.943811    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-kube-api-access-8fd8w\" (UniqueName: \"kubernetes.io/projected/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-kube-api-access-8fd8w\") pod \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\" (UID: \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\") "
Nov 02 02:07:11.944046 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.943879    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-plugins-config-volume\") pod \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\" (UID: \"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79\") "
Nov 02 02:07:11.944321 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.944296    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-plugins-config-volume" pod "b05adf78-f5e8-45dd-87a0-fd8c04cb1c79" (UID: "b05adf78-f5e8-45dd-87a0-fd8c04cb1c79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:11.944596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.944574    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:11.946289 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:11.946267    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-kube-api-access-8fd8w" pod "b05adf78-f5e8-45dd-87a0-fd8c04cb1c79" (UID: "b05adf78-f5e8-45dd-87a0-fd8c04cb1c79"). InnerVolumeSpecName "kube-api-access-8fd8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:12.046560 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.046512    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fd8w\" (UniqueName: \"kubernetes.io/projected/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79-kube-api-access-8fd8w\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:12.138753 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:07:12.138710    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:07:12.145727 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.145698    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:12.345585 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.345549    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79" podUID="2cd02b17-4d56-4d3c-b93b-71278a907bca" containerName="payload-pre-processing" containerID="cri-o://502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba" gracePeriod=25
Nov 02 02:07:12.365845 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.365809    4912 generic.go:356] "Generic (PLEG): container finished" podID="b05adf78-f5e8-45dd-87a0-fd8c04cb1c79" containerID="201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0" exitCode=0
Nov 02 02:07:12.366021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.365885    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns" event={"ID":"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79","Type":"ContainerDied","Data":"201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0"}
Nov 02 02:07:12.366021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.365949    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns" event={"ID":"b05adf78-f5e8-45dd-87a0-fd8c04cb1c79","Type":"ContainerDied","Data":"a3bd1730f3c718f507335dafb56284a104e4da68a32a7ae55b9430db34590465"}
Nov 02 02:07:12.366021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.365892    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"
Nov 02 02:07:12.366232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.365971    4912 scope.go:122] "RemoveContainer" containerID="201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0"
Nov 02 02:07:12.462470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.462446    4912 scope.go:122] "RemoveContainer" containerID="201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0"
Nov 02 02:07:12.462871 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:07:12.462849    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0\": container with ID starting with 201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0 not found: ID does not exist" containerID="201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0"
Nov 02 02:07:12.462939 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.462886    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0"} err="failed to get container status \"201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0\": rpc error: code = NotFound desc = could not find container \"201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0\": container with ID starting with 201a2da8f1c93490f6f54403de7b018e2f25c60d732b837b1275fd6c74c971e0 not found: ID does not exist"
Nov 02 02:07:12.545105 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.541898    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"]
Nov 02 02:07:12.591143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.591110    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-73e1cc-6c7c77ccb7-s2jns"]
Nov 02 02:07:12.629112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.629085    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:07:12.756417 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.756334    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd02b17-4d56-4d3c-b93b-71278a907bca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd02b17-4d56-4d3c-b93b-71278a907bca-plugins-config-volume\") pod \"2cd02b17-4d56-4d3c-b93b-71278a907bca\" (UID: \"2cd02b17-4d56-4d3c-b93b-71278a907bca\") "
Nov 02 02:07:12.756608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.756423    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd02b17-4d56-4d3c-b93b-71278a907bca-kube-api-access-2flnt\" (UniqueName: \"kubernetes.io/projected/2cd02b17-4d56-4d3c-b93b-71278a907bca-kube-api-access-2flnt\") pod \"2cd02b17-4d56-4d3c-b93b-71278a907bca\" (UID: \"2cd02b17-4d56-4d3c-b93b-71278a907bca\") "
Nov 02 02:07:12.756788 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.756759    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd02b17-4d56-4d3c-b93b-71278a907bca-plugins-config-volume" pod "2cd02b17-4d56-4d3c-b93b-71278a907bca" (UID: "2cd02b17-4d56-4d3c-b93b-71278a907bca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:12.757050 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.757033    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd02b17-4d56-4d3c-b93b-71278a907bca-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:12.758866 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.758840    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd02b17-4d56-4d3c-b93b-71278a907bca-kube-api-access-2flnt" pod "2cd02b17-4d56-4d3c-b93b-71278a907bca" (UID: "2cd02b17-4d56-4d3c-b93b-71278a907bca"). InnerVolumeSpecName "kube-api-access-2flnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:12.858536 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:12.858479    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2flnt\" (UniqueName: \"kubernetes.io/projected/2cd02b17-4d56-4d3c-b93b-71278a907bca-kube-api-access-2flnt\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:13.228024 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.227991    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d50c81-35f7-4504-924c-6419fac220c0" path="/var/lib/kubelet/pods/63d50c81-35f7-4504-924c-6419fac220c0/volumes"
Nov 02 02:07:13.228462 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.228448    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05adf78-f5e8-45dd-87a0-fd8c04cb1c79" path="/var/lib/kubelet/pods/b05adf78-f5e8-45dd-87a0-fd8c04cb1c79/volumes"
Nov 02 02:07:13.277020 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.276990    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:07:13.277241 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.277090    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:07:13.378312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.378276    4912 generic.go:356] "Generic (PLEG): container finished" podID="2cd02b17-4d56-4d3c-b93b-71278a907bca" containerID="502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba" exitCode=0
Nov 02 02:07:13.378493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.378337    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"
Nov 02 02:07:13.378493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.378367    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79" event={"ID":"2cd02b17-4d56-4d3c-b93b-71278a907bca","Type":"ContainerDied","Data":"502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba"}
Nov 02 02:07:13.378493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.378424    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79" event={"ID":"2cd02b17-4d56-4d3c-b93b-71278a907bca","Type":"ContainerDied","Data":"d88105ad05291d59cd3e1fba8e2c00c0441b63e822e4317ad94a9b2c9e921083"}
Nov 02 02:07:13.378493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.378448    4912 scope.go:122] "RemoveContainer" containerID="502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba"
Nov 02 02:07:13.393003 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.392979    4912 scope.go:122] "RemoveContainer" containerID="502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba"
Nov 02 02:07:13.393377 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:07:13.393350    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba\": container with ID starting with 502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba not found: ID does not exist" containerID="502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba"
Nov 02 02:07:13.393448 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.393387    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba"} err="failed to get container status \"502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba\": rpc error: code = NotFound desc = could not find container \"502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba\": container with ID starting with 502ca1e598cabfec7d6e6b9c181d2e3d715abcf72c47e364e4929194d08df2ba not found: ID does not exist"
Nov 02 02:07:13.629997 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.629966    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"]
Nov 02 02:07:13.634345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.634325    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa1597-57dddcbc9-npn79"]
Nov 02 02:07:13.872733 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:13.872687    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:15.228031 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:15.228001    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd02b17-4d56-4d3c-b93b-71278a907bca" path="/var/lib/kubelet/pods/2cd02b17-4d56-4d3c-b93b-71278a907bca/volumes"
Nov 02 02:07:18.138154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:18.138117    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:18.337769 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:18.337735    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:18.508180 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:18.508091    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:18.714747 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:18.713140    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:18.892872 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:18.892578    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:22.587010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.586973    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"]
Nov 02 02:07:22.587741 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.587719    4912 cpu_manager.go:414] "containerMap: removing container" podUID="2cd02b17-4d56-4d3c-b93b-71278a907bca" containerName="payload-pre-processing"
Nov 02 02:07:22.587818 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.587743    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd02b17-4d56-4d3c-b93b-71278a907bca" containerName="payload-pre-processing"
Nov 02 02:07:22.587818 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.587764    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b7d15873-c164-4c41-aac9-bff2fa0f72fe" containerName="maas-api"
Nov 02 02:07:22.587818 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.587774    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d15873-c164-4c41-aac9-bff2fa0f72fe" containerName="maas-api"
Nov 02 02:07:22.587982 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.587820    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b05adf78-f5e8-45dd-87a0-fd8c04cb1c79" containerName="payload-pre-processing"
Nov 02 02:07:22.587982 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.587828    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05adf78-f5e8-45dd-87a0-fd8c04cb1c79" containerName="payload-pre-processing"
Nov 02 02:07:22.588145 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.588039    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd02b17-4d56-4d3c-b93b-71278a907bca" containerName="payload-pre-processing"
Nov 02 02:07:22.588145 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.588080    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d15873-c164-4c41-aac9-bff2fa0f72fe" containerName="maas-api"
Nov 02 02:07:22.588145 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.588095    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05adf78-f5e8-45dd-87a0-fd8c04cb1c79" containerName="payload-pre-processing"
Nov 02 02:07:22.593240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.593219    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.649727 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.649692    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"]
Nov 02 02:07:22.692286 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.692251    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:07:22.692495 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.692455    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:07:22.708280 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.708249    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.708433 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.708407    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t464t\" (UniqueName: \"kubernetes.io/projected/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-kube-api-access-t464t\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.708550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.708530    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.809317 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.809278    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.809521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.809363    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.809521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.809468    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t464t\" (UniqueName: \"kubernetes.io/projected/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-kube-api-access-t464t\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.809993 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.809965    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.811873 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.811851    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.870261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.870222    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t464t\" (UniqueName: \"kubernetes.io/projected/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-kube-api-access-t464t\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:22.909530 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:22.909495    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:23.188134 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:23.185693    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:23.303700 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:07:23.302811    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e90a7da_f4fc_4ebc_994f_c6a22dc023f0.slice/crio-23a4c487eb25f391dda0276b1e2343df6b56f38c4e6d62052b8320bc9cf4495a WatchSource:0}: Error finding container 23a4c487eb25f391dda0276b1e2343df6b56f38c4e6d62052b8320bc9cf4495a: Status 404 returned error can't find the container with id 23a4c487eb25f391dda0276b1e2343df6b56f38c4e6d62052b8320bc9cf4495a
Nov 02 02:07:23.303700 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:23.302853    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"]
Nov 02 02:07:23.561132 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:23.561091    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q" event={"ID":"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0","Type":"ContainerStarted","Data":"23a4c487eb25f391dda0276b1e2343df6b56f38c4e6d62052b8320bc9cf4495a"}
Nov 02 02:07:24.574816 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:24.574715    4912 generic.go:356] "Generic (PLEG): container finished" podID="0e90a7da-f4fc-4ebc-994f-c6a22dc023f0" containerID="7d5e3f9de6c1988ceccb243c165e65cfa64097976e5d5821a9dbccf603fc4731" exitCode=0
Nov 02 02:07:24.574816 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:24.574791    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q" event={"ID":"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0","Type":"ContainerDied","Data":"7d5e3f9de6c1988ceccb243c165e65cfa64097976e5d5821a9dbccf603fc4731"}
Nov 02 02:07:24.817353 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:24.817318    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:25.082463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.082425    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:25.259759 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.259717    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:25.323685 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.323645    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"]
Nov 02 02:07:25.328431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.328393    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc85a-hc7tt"]
Nov 02 02:07:25.761354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.761330    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:25.796924 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.796890    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-xks-ca\") pod \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") "
Nov 02 02:07:25.797171 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.796983    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-kube-api-access-t464t\" (UniqueName: \"kubernetes.io/projected/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-kube-api-access-t464t\") pod \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") "
Nov 02 02:07:25.797251 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.797191    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-openshift-ca\") pod \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\" (UID: \"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0\") "
Nov 02 02:07:25.797520 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.797490    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-openshift-ca" pod "0e90a7da-f4fc-4ebc-994f-c6a22dc023f0" (UID: "0e90a7da-f4fc-4ebc-994f-c6a22dc023f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:25.797700 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.797681    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.799367 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.799339    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-xks-ca" pod "0e90a7da-f4fc-4ebc-994f-c6a22dc023f0" (UID: "0e90a7da-f4fc-4ebc-994f-c6a22dc023f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:25.799467 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.799446    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-kube-api-access-t464t" pod "0e90a7da-f4fc-4ebc-994f-c6a22dc023f0" (UID: "0e90a7da-f4fc-4ebc-994f-c6a22dc023f0"). InnerVolumeSpecName "kube-api-access-t464t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:25.899551 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.899508    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.899551 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:25.899553    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t464t\" (UniqueName: \"kubernetes.io/projected/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0-kube-api-access-t464t\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:26.174271 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:26.174182    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:26.595101 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:26.595004    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"
Nov 02 02:07:26.595101 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:26.595017    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q" event={"ID":"0e90a7da-f4fc-4ebc-994f-c6a22dc023f0","Type":"ContainerDied","Data":"23a4c487eb25f391dda0276b1e2343df6b56f38c4e6d62052b8320bc9cf4495a"}
Nov 02 02:07:26.595101 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:26.595093    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23a4c487eb25f391dda0276b1e2343df6b56f38c4e6d62052b8320bc9cf4495a"
Nov 02 02:07:27.238497 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:27.238456    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbcf169-58cd-40a7-83e0-103a2e1c8f6c" path="/var/lib/kubelet/pods/bdbcf169-58cd-40a7-83e0-103a2e1c8f6c/volumes"
Nov 02 02:07:29.567886 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:29.567836    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:30.800429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:30.800392    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:31.924345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:31.924285    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:32.200819 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:32.200725    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:32.390934 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:32.390902    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"]
Nov 02 02:07:32.393876 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:32.393838    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:32.440516 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:32.440484    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-73e1cc-qt52d"]
Nov 02 02:07:32.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:32.571975    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:32.755313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:32.755261    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:32.956342 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:32.956305    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:33.148073 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:33.147866    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:33.229144 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:33.229044    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f90746-00e6-45fa-89a7-c7a5ade39f7f" path="/var/lib/kubelet/pods/69f90746-00e6-45fa-89a7-c7a5ade39f7f/volumes"
Nov 02 02:07:33.328731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:33.328693    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:33.552869 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:33.552792    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"]
Nov 02 02:07:33.601705 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:33.601674    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-129de4-fgnsv"]
Nov 02 02:07:34.222901 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:34.222870    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:34.875007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:34.874976    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"]
Nov 02 02:07:35.227451 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.227367    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d66d8b4d-fa84-4a75-b850-e2197402a346" path="/var/lib/kubelet/pods/d66d8b4d-fa84-4a75-b850-e2197402a346/volumes"
Nov 02 02:07:35.342013 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.341985    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:35.578977 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.578895    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:35.920885 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.920850    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"]
Nov 02 02:07:35.921571 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.921556    4912 cpu_manager.go:414] "containerMap: removing container" podUID="0e90a7da-f4fc-4ebc-994f-c6a22dc023f0" containerName="revoke-keys"
Nov 02 02:07:35.921652 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.921572    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e90a7da-f4fc-4ebc-994f-c6a22dc023f0" containerName="revoke-keys"
Nov 02 02:07:35.921726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.921716    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e90a7da-f4fc-4ebc-994f-c6a22dc023f0" containerName="revoke-keys"
Nov 02 02:07:35.926795 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.926777    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:35.979799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.979768    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:07:35.980087 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.980043    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:07:35.982440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:35.982409    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"]
Nov 02 02:07:36.086127 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.086090    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s528w\" (UniqueName: \"kubernetes.io/projected/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-kube-api-access-s528w\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.086338 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.086316    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.086432 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.086416    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.187793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.187708    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.187946 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.187800    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.187946 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.187915    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s528w\" (UniqueName: \"kubernetes.io/projected/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-kube-api-access-s528w\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.188456 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.188426    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.190670 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.190639    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.248630 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.248598    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s528w\" (UniqueName: \"kubernetes.io/projected/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-kube-api-access-s528w\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.251445 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.251425    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:36.717879 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:07:36.717846    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5578ff_6c42_44a3_b4b2_95fac1a297b9.slice/crio-ff62d7211ed45fa60012ae5d7ff043637cca095a7e820f21c2ce0bd43fdf6ec0 WatchSource:0}: Error finding container ff62d7211ed45fa60012ae5d7ff043637cca095a7e820f21c2ce0bd43fdf6ec0: Status 404 returned error can't find the container with id ff62d7211ed45fa60012ae5d7ff043637cca095a7e820f21c2ce0bd43fdf6ec0
Nov 02 02:07:36.718658 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:36.718627    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"]
Nov 02 02:07:37.718639 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:37.718603    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw" event={"ID":"3a5578ff-6c42-44a3-b4b2-95fac1a297b9","Type":"ContainerStarted","Data":"318a03386fce85b201edd11217acac96a5076cb56836ac66e99c145eca416284"}
Nov 02 02:07:37.719118 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:37.718670    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw" event={"ID":"3a5578ff-6c42-44a3-b4b2-95fac1a297b9","Type":"ContainerStarted","Data":"ff62d7211ed45fa60012ae5d7ff043637cca095a7e820f21c2ce0bd43fdf6ec0"}
Nov 02 02:07:37.835368 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:37.835316    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw" podStartSLOduration=2.8353027170000002 podStartE2EDuration="2.835302717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:37.833972563 +0000 UTC m=+2150.739437921" watchObservedRunningTime="2026-11-02 02:07:37.835302717 +0000 UTC m=+2150.740768076"
Nov 02 02:07:37.852829 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:37.852796    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:38.728960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:38.728914    4912 generic.go:356] "Generic (PLEG): container finished" podID="3a5578ff-6c42-44a3-b4b2-95fac1a297b9" containerID="318a03386fce85b201edd11217acac96a5076cb56836ac66e99c145eca416284" exitCode=0
Nov 02 02:07:38.729422 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:38.728977    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw" event={"ID":"3a5578ff-6c42-44a3-b4b2-95fac1a297b9","Type":"ContainerDied","Data":"318a03386fce85b201edd11217acac96a5076cb56836ac66e99c145eca416284"}
Nov 02 02:07:39.144166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:39.144127    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:39.379041 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:39.379000    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:39.601959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:39.601883    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:39.875569 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:39.875342    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h" podUID="167482e0-1d23-4b89-96af-369f536338e9" containerName="payload-processing" containerID="cri-o://03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2" gracePeriod=25
Nov 02 02:07:40.021121 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.021089    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:40.114431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.108518    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-openshift-ca\") pod \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") "
Nov 02 02:07:40.114431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.108595    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-kube-api-access-s528w\" (UniqueName: \"kubernetes.io/projected/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-kube-api-access-s528w\") pod \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") "
Nov 02 02:07:40.114431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.108641    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-xks-ca\") pod \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\" (UID: \"3a5578ff-6c42-44a3-b4b2-95fac1a297b9\") "
Nov 02 02:07:40.115150 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.114816    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-xks-ca" pod "3a5578ff-6c42-44a3-b4b2-95fac1a297b9" (UID: "3a5578ff-6c42-44a3-b4b2-95fac1a297b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:40.118231 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.118206    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-kube-api-access-s528w" pod "3a5578ff-6c42-44a3-b4b2-95fac1a297b9" (UID: "3a5578ff-6c42-44a3-b4b2-95fac1a297b9"). InnerVolumeSpecName "kube-api-access-s528w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:40.119308 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.119283    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-openshift-ca" pod "3a5578ff-6c42-44a3-b4b2-95fac1a297b9" (UID: "3a5578ff-6c42-44a3-b4b2-95fac1a297b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:40.164772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.164740    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 02:07:40.210552 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.210515    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/167482e0-1d23-4b89-96af-369f536338e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167482e0-1d23-4b89-96af-369f536338e9-plugins-config-volume\") pod \"167482e0-1d23-4b89-96af-369f536338e9\" (UID: \"167482e0-1d23-4b89-96af-369f536338e9\") "
Nov 02 02:07:40.210776 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.210598    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167482e0-1d23-4b89-96af-369f536338e9-kube-api-access-l6xhg\" (UniqueName: \"kubernetes.io/projected/167482e0-1d23-4b89-96af-369f536338e9-kube-api-access-l6xhg\") pod \"167482e0-1d23-4b89-96af-369f536338e9\" (UID: \"167482e0-1d23-4b89-96af-369f536338e9\") "
Nov 02 02:07:40.210927 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.210903    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167482e0-1d23-4b89-96af-369f536338e9-plugins-config-volume" pod "167482e0-1d23-4b89-96af-369f536338e9" (UID: "167482e0-1d23-4b89-96af-369f536338e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:40.211337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.211314    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:40.211514 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.211341    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s528w\" (UniqueName: \"kubernetes.io/projected/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-kube-api-access-s528w\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:40.211514 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.211370    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5578ff-6c42-44a3-b4b2-95fac1a297b9-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:40.211514 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.211384    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167482e0-1d23-4b89-96af-369f536338e9-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:40.213397 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.213371    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167482e0-1d23-4b89-96af-369f536338e9-kube-api-access-l6xhg" pod "167482e0-1d23-4b89-96af-369f536338e9" (UID: "167482e0-1d23-4b89-96af-369f536338e9"). InnerVolumeSpecName "kube-api-access-l6xhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:40.313405 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.313363    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6xhg\" (UniqueName: \"kubernetes.io/projected/167482e0-1d23-4b89-96af-369f536338e9-kube-api-access-l6xhg\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:40.334442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.334400    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"]
Nov 02 02:07:40.492169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.492085    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"]
Nov 02 02:07:40.493108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.493085    4912 cpu_manager.go:414] "containerMap: removing container" podUID="3a5578ff-6c42-44a3-b4b2-95fac1a297b9" containerName="revoke-keys"
Nov 02 02:07:40.493108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.493106    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5578ff-6c42-44a3-b4b2-95fac1a297b9" containerName="revoke-keys"
Nov 02 02:07:40.493272 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.493121    4912 cpu_manager.go:414] "containerMap: removing container" podUID="167482e0-1d23-4b89-96af-369f536338e9" containerName="payload-processing"
Nov 02 02:07:40.493272 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.493127    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167482e0-1d23-4b89-96af-369f536338e9" containerName="payload-processing"
Nov 02 02:07:40.493343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.493271    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="167482e0-1d23-4b89-96af-369f536338e9" containerName="payload-processing"
Nov 02 02:07:40.493343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.493291    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5578ff-6c42-44a3-b4b2-95fac1a297b9" containerName="revoke-keys"
Nov 02 02:07:40.497190 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.497164    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.497926 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.497888    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:40.516620 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.516592    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130d495f-9c80-4c13-8577-aaede795f977-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.516767 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.516645    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130d495f-9c80-4c13-8577-aaede795f977-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.516836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.516821    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tprld\" (UniqueName: \"kubernetes.io/projected/130d495f-9c80-4c13-8577-aaede795f977-kube-api-access-tprld\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.573110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.562545    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"]
Nov 02 02:07:40.618374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.618336    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130d495f-9c80-4c13-8577-aaede795f977-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.618574 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.618552    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tprld\" (UniqueName: \"kubernetes.io/projected/130d495f-9c80-4c13-8577-aaede795f977-kube-api-access-tprld\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.618730 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.618712    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130d495f-9c80-4c13-8577-aaede795f977-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.619514 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.619478    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130d495f-9c80-4c13-8577-aaede795f977-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.621123 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.621098    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130d495f-9c80-4c13-8577-aaede795f977-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.679132 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.679082    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tprld\" (UniqueName: \"kubernetes.io/projected/130d495f-9c80-4c13-8577-aaede795f977-kube-api-access-tprld\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.814150 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.814023    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:40.817952 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.817921    4912 generic.go:356] "Generic (PLEG): container finished" podID="167482e0-1d23-4b89-96af-369f536338e9" containerID="03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2" exitCode=0
Nov 02 02:07:40.818137 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.817971    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h" event={"ID":"167482e0-1d23-4b89-96af-369f536338e9","Type":"ContainerDied","Data":"03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2"}
Nov 02 02:07:40.818137 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.818031    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h" event={"ID":"167482e0-1d23-4b89-96af-369f536338e9","Type":"ContainerDied","Data":"97fd7b14dc9d2193e51166cbee551f616271a6828af5fdc65601fd5a271566d8"}
Nov 02 02:07:40.818137 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.818079    4912 scope.go:122] "RemoveContainer" containerID="03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2"
Nov 02 02:07:40.818337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.818321    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"
Nov 02 02:07:40.819700 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.819682    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"
Nov 02 02:07:40.819792 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.819749    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw" event={"ID":"3a5578ff-6c42-44a3-b4b2-95fac1a297b9","Type":"ContainerDied","Data":"ff62d7211ed45fa60012ae5d7ff043637cca095a7e820f21c2ce0bd43fdf6ec0"}
Nov 02 02:07:40.819850 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.819803    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff62d7211ed45fa60012ae5d7ff043637cca095a7e820f21c2ce0bd43fdf6ec0"
Nov 02 02:07:40.828543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.828513    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:40.833497 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.833474    4912 scope.go:122] "RemoveContainer" containerID="03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2"
Nov 02 02:07:40.833835 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:07:40.833817    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2\": container with ID starting with 03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2 not found: ID does not exist" containerID="03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2"
Nov 02 02:07:40.833918 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.833849    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2"} err="failed to get container status \"03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2\": rpc error: code = NotFound desc = could not find container \"03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2\": container with ID starting with 03af19fe74ad09e7caf759b2fef2ba21c865363a502e2e04daade980ab6bd5a2 not found: ID does not exist"
Nov 02 02:07:40.996193 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:40.996159    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"]
Nov 02 02:07:41.003202 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:41.003171    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-606ca0-55f4dd7d75-r2g7h"]
Nov 02 02:07:41.011763 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:41.011735    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:41.199215 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:41.199175    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:41.292424 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:41.281214    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="167482e0-1d23-4b89-96af-369f536338e9" path="/var/lib/kubelet/pods/167482e0-1d23-4b89-96af-369f536338e9/volumes"
Nov 02 02:07:41.337803 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:41.337771    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"]
Nov 02 02:07:41.832653 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:41.832618    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6" event={"ID":"130d495f-9c80-4c13-8577-aaede795f977","Type":"ContainerStarted","Data":"20bf14cef2e3e2ae7a9b937cdf61bb244d1bb9deb1c7ac4a108fd69057914f22"}
Nov 02 02:07:41.832865 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:41.832675    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6" event={"ID":"130d495f-9c80-4c13-8577-aaede795f977","Type":"ContainerStarted","Data":"80da538fcb1ed0ca4babc510c9475ea4e459fee5f619a65f4676545cce51e4f3"}
Nov 02 02:07:41.952213 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:41.952085    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6" podStartSLOduration=1.9520495740000001 podStartE2EDuration="1.952049574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:41.947212825 +0000 UTC m=+2154.852678182" watchObservedRunningTime="2026-11-02 02:07:41.952049574 +0000 UTC m=+2154.857514932"
Nov 02 02:07:42.075757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.075724    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"]
Nov 02 02:07:42.080409 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.080389    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.138282 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.138250    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"]
Nov 02 02:07:42.153831 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:07:42.153767    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:07:42.160440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.160411    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8fwk\" (UniqueName: \"kubernetes.io/projected/be329a34-1d02-47c6-87d2-3224449b303c-kube-api-access-v8fwk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.160624 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.160469    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.160694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.160656    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.262194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.262102    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8fwk\" (UniqueName: \"kubernetes.io/projected/be329a34-1d02-47c6-87d2-3224449b303c-kube-api-access-v8fwk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.262194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.262171    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.262442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.262335    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.263168 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.263143    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.265222 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.265199    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.323340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.323308    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8fwk\" (UniqueName: \"kubernetes.io/projected/be329a34-1d02-47c6-87d2-3224449b303c-kube-api-access-v8fwk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.395095 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.395047    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:42.782626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.782591    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"]
Nov 02 02:07:42.783505 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:07:42.783463    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe329a34_1d02_47c6_87d2_3224449b303c.slice/crio-f798522dfeaac3494e7f6d92fd6634920ca36e61090d8f51ca51e313b03370d4 WatchSource:0}: Error finding container f798522dfeaac3494e7f6d92fd6634920ca36e61090d8f51ca51e313b03370d4: Status 404 returned error can't find the container with id f798522dfeaac3494e7f6d92fd6634920ca36e61090d8f51ca51e313b03370d4
Nov 02 02:07:42.842413 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.842382    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq" event={"ID":"be329a34-1d02-47c6-87d2-3224449b303c","Type":"ContainerStarted","Data":"f798522dfeaac3494e7f6d92fd6634920ca36e61090d8f51ca51e313b03370d4"}
Nov 02 02:07:42.843966 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.843938    4912 generic.go:356] "Generic (PLEG): container finished" podID="130d495f-9c80-4c13-8577-aaede795f977" containerID="20bf14cef2e3e2ae7a9b937cdf61bb244d1bb9deb1c7ac4a108fd69057914f22" exitCode=0
Nov 02 02:07:42.844113 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:42.844020    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6" event={"ID":"130d495f-9c80-4c13-8577-aaede795f977","Type":"ContainerDied","Data":"20bf14cef2e3e2ae7a9b937cdf61bb244d1bb9deb1c7ac4a108fd69057914f22"}
Nov 02 02:07:43.145564 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:43.145526    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:43.854672 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:43.854631    4912 generic.go:356] "Generic (PLEG): container finished" podID="be329a34-1d02-47c6-87d2-3224449b303c" containerID="a8cbf4cff3fa13d6d0b0828405de51abfa9338780529fe03e033b69806134614" exitCode=0
Nov 02 02:07:43.854890 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:43.854745    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq" event={"ID":"be329a34-1d02-47c6-87d2-3224449b303c","Type":"ContainerDied","Data":"a8cbf4cff3fa13d6d0b0828405de51abfa9338780529fe03e033b69806134614"}
Nov 02 02:07:44.032774 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.032747    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:44.193670 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.193634    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130d495f-9c80-4c13-8577-aaede795f977-kube-api-access-tprld\" (UniqueName: \"kubernetes.io/projected/130d495f-9c80-4c13-8577-aaede795f977-kube-api-access-tprld\") pod \"130d495f-9c80-4c13-8577-aaede795f977\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") "
Nov 02 02:07:44.196543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.193755    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/130d495f-9c80-4c13-8577-aaede795f977-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130d495f-9c80-4c13-8577-aaede795f977-maas-api-xks-ca\") pod \"130d495f-9c80-4c13-8577-aaede795f977\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") "
Nov 02 02:07:44.196543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.193813    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/130d495f-9c80-4c13-8577-aaede795f977-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130d495f-9c80-4c13-8577-aaede795f977-maas-api-openshift-ca\") pod \"130d495f-9c80-4c13-8577-aaede795f977\" (UID: \"130d495f-9c80-4c13-8577-aaede795f977\") "
Nov 02 02:07:44.196543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.194636    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130d495f-9c80-4c13-8577-aaede795f977-maas-api-openshift-ca" pod "130d495f-9c80-4c13-8577-aaede795f977" (UID: "130d495f-9c80-4c13-8577-aaede795f977"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:44.196744 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.196719    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130d495f-9c80-4c13-8577-aaede795f977-maas-api-xks-ca" pod "130d495f-9c80-4c13-8577-aaede795f977" (UID: "130d495f-9c80-4c13-8577-aaede795f977"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:44.200392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.200360    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130d495f-9c80-4c13-8577-aaede795f977-kube-api-access-tprld" pod "130d495f-9c80-4c13-8577-aaede795f977" (UID: "130d495f-9c80-4c13-8577-aaede795f977"). InnerVolumeSpecName "kube-api-access-tprld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:44.296035 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.295936    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tprld\" (UniqueName: \"kubernetes.io/projected/130d495f-9c80-4c13-8577-aaede795f977-kube-api-access-tprld\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:44.296035 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.295974    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130d495f-9c80-4c13-8577-aaede795f977-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:44.296035 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.295988    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130d495f-9c80-4c13-8577-aaede795f977-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:44.865811 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.865780    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"
Nov 02 02:07:44.865811 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.865799    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6" event={"ID":"130d495f-9c80-4c13-8577-aaede795f977","Type":"ContainerDied","Data":"80da538fcb1ed0ca4babc510c9475ea4e459fee5f619a65f4676545cce51e4f3"}
Nov 02 02:07:44.866079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:44.865858    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80da538fcb1ed0ca4babc510c9475ea4e459fee5f619a65f4676545cce51e4f3"
Nov 02 02:07:45.038911 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.038870    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:45.208261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.208170    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-openshift-ca\") pod \"be329a34-1d02-47c6-87d2-3224449b303c\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") "
Nov 02 02:07:45.208261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.208262    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-xks-ca\") pod \"be329a34-1d02-47c6-87d2-3224449b303c\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") "
Nov 02 02:07:45.208790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.208355    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be329a34-1d02-47c6-87d2-3224449b303c-kube-api-access-v8fwk\" (UniqueName: \"kubernetes.io/projected/be329a34-1d02-47c6-87d2-3224449b303c-kube-api-access-v8fwk\") pod \"be329a34-1d02-47c6-87d2-3224449b303c\" (UID: \"be329a34-1d02-47c6-87d2-3224449b303c\") "
Nov 02 02:07:45.208790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.208497    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-openshift-ca" pod "be329a34-1d02-47c6-87d2-3224449b303c" (UID: "be329a34-1d02-47c6-87d2-3224449b303c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:45.208976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.208957    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:45.210644 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.210623    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be329a34-1d02-47c6-87d2-3224449b303c-kube-api-access-v8fwk" pod "be329a34-1d02-47c6-87d2-3224449b303c" (UID: "be329a34-1d02-47c6-87d2-3224449b303c"). InnerVolumeSpecName "kube-api-access-v8fwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:45.210734 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.210664    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-xks-ca" pod "be329a34-1d02-47c6-87d2-3224449b303c" (UID: "be329a34-1d02-47c6-87d2-3224449b303c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:45.310365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.310316    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8fwk\" (UniqueName: \"kubernetes.io/projected/be329a34-1d02-47c6-87d2-3224449b303c-kube-api-access-v8fwk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:45.310365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.310352    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be329a34-1d02-47c6-87d2-3224449b303c-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:45.877127 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.877097    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"
Nov 02 02:07:45.877312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.877089    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq" event={"ID":"be329a34-1d02-47c6-87d2-3224449b303c","Type":"ContainerDied","Data":"f798522dfeaac3494e7f6d92fd6634920ca36e61090d8f51ca51e313b03370d4"}
Nov 02 02:07:45.877312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:45.877251    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f798522dfeaac3494e7f6d92fd6634920ca36e61090d8f51ca51e313b03370d4"
Nov 02 02:07:47.523740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:47.523644    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"]
Nov 02 02:07:47.546830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:47.546783    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:47.806397 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:47.806298    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:47.984947 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:47.984909    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:49.233085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:49.232421    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:49.490032 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:49.489927    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:49.673109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:49.672918    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:49.700939 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:49.700903    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"]
Nov 02 02:07:49.747391 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:49.747317    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c1aad5-rrz9k"]
Nov 02 02:07:50.159907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:50.159872    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:50.339689 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:50.339652    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:51.227470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:51.227438    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0d1913-f26e-43b9-a36f-04702cd3e560" path="/var/lib/kubelet/pods/9c0d1913-f26e-43b9-a36f-04702cd3e560/volumes"
Nov 02 02:07:52.379313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:52.379278    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:52.525565 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:52.525504    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk" podUID="2fa22154-c3f8-47d0-8523-109b05e3a993" containerName="payload-pre-processing" containerID="cri-o://997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c" gracePeriod=25
Nov 02 02:07:52.642683 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:52.641972    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:54.637617 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:54.637581    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:54.885784 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:54.885747    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:55.060329 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.060233    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:55.357541 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.357449    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main" containerID="cri-o://e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9" gracePeriod=45
Nov 02 02:07:55.645261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.645220    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 02:07:55.729180 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.729147    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk" podUID="2fa22154-c3f8-47d0-8523-109b05e3a993" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:07:55.759149 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.759109    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tmp-dir\") pod \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") "
Nov 02 02:07:55.759318 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.759228    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6226fcd-61ac-4173-a1fa-28378df2bb1b-kube-api-access-hlwvt\" (UniqueName: \"kubernetes.io/projected/b6226fcd-61ac-4173-a1fa-28378df2bb1b-kube-api-access-hlwvt\") pod \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") "
Nov 02 02:07:55.759318 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.759284    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-dshm\") pod \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") "
Nov 02 02:07:55.759428 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.759407    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tmp-dir" pod "b6226fcd-61ac-4173-a1fa-28378df2bb1b" (UID: "b6226fcd-61ac-4173-a1fa-28378df2bb1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:55.759510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.759478    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tls-certs\") pod \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") "
Nov 02 02:07:55.759661 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.759646    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-home\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-home\") pod \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") "
Nov 02 02:07:55.759722 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.759694    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-model-cache\") pod \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\" (UID: \"b6226fcd-61ac-4173-a1fa-28378df2bb1b\") "
Nov 02 02:07:55.760143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.759938    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-home" pod "b6226fcd-61ac-4173-a1fa-28378df2bb1b" (UID: "b6226fcd-61ac-4173-a1fa-28378df2bb1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:55.760143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.759976    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-model-cache" pod "b6226fcd-61ac-4173-a1fa-28378df2bb1b" (UID: "b6226fcd-61ac-4173-a1fa-28378df2bb1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:55.760657 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.760634    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.760783 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.760663    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.760783 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.760706    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.762393 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.762357    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-dshm" pod "b6226fcd-61ac-4173-a1fa-28378df2bb1b" (UID: "b6226fcd-61ac-4173-a1fa-28378df2bb1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:55.762831 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.762802    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tls-certs" pod "b6226fcd-61ac-4173-a1fa-28378df2bb1b" (UID: "b6226fcd-61ac-4173-a1fa-28378df2bb1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:55.762931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.762856    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6226fcd-61ac-4173-a1fa-28378df2bb1b-kube-api-access-hlwvt" pod "b6226fcd-61ac-4173-a1fa-28378df2bb1b" (UID: "b6226fcd-61ac-4173-a1fa-28378df2bb1b"). InnerVolumeSpecName "kube-api-access-hlwvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:55.862561 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.862478    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlwvt\" (UniqueName: \"kubernetes.io/projected/b6226fcd-61ac-4173-a1fa-28378df2bb1b-kube-api-access-hlwvt\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.862561 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.862507    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6226fcd-61ac-4173-a1fa-28378df2bb1b-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.862561 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:55.862518    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6226fcd-61ac-4173-a1fa-28378df2bb1b-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:56.013660 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.013623    4912 generic.go:356] "Generic (PLEG): container finished" podID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerID="e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9" exitCode=0
Nov 02 02:07:56.013882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.013666    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"
Nov 02 02:07:56.013882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.013709    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" event={"ID":"b6226fcd-61ac-4173-a1fa-28378df2bb1b","Type":"ContainerDied","Data":"e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9"}
Nov 02 02:07:56.013882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.013755    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f" event={"ID":"b6226fcd-61ac-4173-a1fa-28378df2bb1b","Type":"ContainerDied","Data":"721e236cbf2260206088f6b844cff08c1c0ea15ecb6b602b8d3ad42f5d66364f"}
Nov 02 02:07:56.013882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.013776    4912 scope.go:122] "RemoveContainer" containerID="e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9"
Nov 02 02:07:56.119469 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.110245    4912 scope.go:122] "RemoveContainer" containerID="e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9"
Nov 02 02:07:56.119469 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:07:56.110680    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9\": container with ID starting with e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9 not found: ID does not exist" containerID="e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9"
Nov 02 02:07:56.119469 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.110720    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9"} err="failed to get container status \"e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9\": rpc error: code = NotFound desc = could not find container \"e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9\": container with ID starting with e8379b165ef73d13d4f7b0b45cabf3be5482d3bfcdf995fab384b6c15a3acdf9 not found: ID does not exist"
Nov 02 02:07:56.186188 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.186143    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"]
Nov 02 02:07:56.245091 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.244296    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/distinct-2-w4-ed0600-kserve-76c7999d9-njm6f"]
Nov 02 02:07:56.607834 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.607757    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:56.841448 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.841411    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:56.987574 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.987536    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"]
Nov 02 02:07:56.987805 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:56.987781    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy" containerID="cri-o://58cdae4d2708055b31e394ca340212be792c95420df3eebe073f630f460a2164" gracePeriod=30
Nov 02 02:07:57.020556 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.020525    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:57.229251 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.229217    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" path="/var/lib/kubelet/pods/b6226fcd-61ac-4173-a1fa-28378df2bb1b/volumes"
Nov 02 02:07:57.962972 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.962932    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"]
Nov 02 02:07:57.963843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.963822    4912 cpu_manager.go:414] "containerMap: removing container" podUID="130d495f-9c80-4c13-8577-aaede795f977" containerName="revoke-keys"
Nov 02 02:07:57.963972 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.963846    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130d495f-9c80-4c13-8577-aaede795f977" containerName="revoke-keys"
Nov 02 02:07:57.963972 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.963891    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main"
Nov 02 02:07:57.963972 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.963899    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main"
Nov 02 02:07:57.963972 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.963929    4912 cpu_manager.go:414] "containerMap: removing container" podUID="be329a34-1d02-47c6-87d2-3224449b303c" containerName="revoke-keys"
Nov 02 02:07:57.963972 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.963937    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be329a34-1d02-47c6-87d2-3224449b303c" containerName="revoke-keys"
Nov 02 02:07:57.964283 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.964165    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6226fcd-61ac-4173-a1fa-28378df2bb1b" containerName="main"
Nov 02 02:07:57.964283 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.964183    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="be329a34-1d02-47c6-87d2-3224449b303c" containerName="revoke-keys"
Nov 02 02:07:57.964283 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.964201    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="130d495f-9c80-4c13-8577-aaede795f977" containerName="revoke-keys"
Nov 02 02:07:57.969431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:57.969411    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.025560 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.025523    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"]
Nov 02 02:07:58.068580 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.068551    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:07:58.068580 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.068552    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:07:58.103231 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.103197    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.103419 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.103319    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.103509 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.103481    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59m5\" (UniqueName: \"kubernetes.io/projected/9e7613f4-bddc-450d-aa39-2f88e37759e5-kube-api-access-f59m5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.205191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.205151    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59m5\" (UniqueName: \"kubernetes.io/projected/9e7613f4-bddc-450d-aa39-2f88e37759e5-kube-api-access-f59m5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.205403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.205298    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.205403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.205346    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.206052 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.206024    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.207880 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.207855    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.266862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.266783    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59m5\" (UniqueName: \"kubernetes.io/projected/9e7613f4-bddc-450d-aa39-2f88e37759e5-kube-api-access-f59m5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.283950 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.283916    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:07:58.679189 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:58.679157    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"]
Nov 02 02:07:58.679323 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:07:58.679168    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7613f4_bddc_450d_aa39_2f88e37759e5.slice/crio-6105b3dad06d6ad151b6e7369b4058c6b2ceac892b516c7e7a8a739d6ab9e000 WatchSource:0}: Error finding container 6105b3dad06d6ad151b6e7369b4058c6b2ceac892b516c7e7a8a739d6ab9e000: Status 404 returned error can't find the container with id 6105b3dad06d6ad151b6e7369b4058c6b2ceac892b516c7e7a8a739d6ab9e000
Nov 02 02:07:59.006797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.006764    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"]
Nov 02 02:07:59.007271 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.006967    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk" podUID="82976de6-070a-4904-b91c-6f6f0adf8aa7" containerName="maas-api" containerID="cri-o://0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66" gracePeriod=30
Nov 02 02:07:59.043383 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.043338    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8" event={"ID":"9e7613f4-bddc-450d-aa39-2f88e37759e5","Type":"ContainerStarted","Data":"080c32ac169b20aaf4de877148517c6c5342699c42aaf343b382894355a7f06f"}
Nov 02 02:07:59.043569 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.043405    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8" event={"ID":"9e7613f4-bddc-450d-aa39-2f88e37759e5","Type":"ContainerStarted","Data":"6105b3dad06d6ad151b6e7369b4058c6b2ceac892b516c7e7a8a739d6ab9e000"}
Nov 02 02:07:59.136308 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.136271    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:07:59.262862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.262794    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8" podStartSLOduration=2.26275773 podStartE2EDuration="2.26275773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:59.262333695 +0000 UTC m=+2172.167799054" watchObservedRunningTime="2026-11-02 02:07:59.26275773 +0000 UTC m=+2172.168223088"
Nov 02 02:07:59.347499 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.347467    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 02:07:59.430083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.426676    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls\") pod \"82976de6-070a-4904-b91c-6f6f0adf8aa7\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") "
Nov 02 02:07:59.430083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.426754    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82976de6-070a-4904-b91c-6f6f0adf8aa7-kube-api-access-vwn5c\" (UniqueName: \"kubernetes.io/projected/82976de6-070a-4904-b91c-6f6f0adf8aa7-kube-api-access-vwn5c\") pod \"82976de6-070a-4904-b91c-6f6f0adf8aa7\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") "
Nov 02 02:07:59.430083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.426861    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-metrics-certs\") pod \"82976de6-070a-4904-b91c-6f6f0adf8aa7\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") "
Nov 02 02:07:59.430083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.427007    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82976de6-070a-4904-b91c-6f6f0adf8aa7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82976de6-070a-4904-b91c-6f6f0adf8aa7-openshift-service-ca\") pod \"82976de6-070a-4904-b91c-6f6f0adf8aa7\" (UID: \"82976de6-070a-4904-b91c-6f6f0adf8aa7\") "
Nov 02 02:07:59.430506 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.430243    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82976de6-070a-4904-b91c-6f6f0adf8aa7-openshift-service-ca" pod "82976de6-070a-4904-b91c-6f6f0adf8aa7" (UID: "82976de6-070a-4904-b91c-6f6f0adf8aa7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:59.438934 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.438870    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls" pod "82976de6-070a-4904-b91c-6f6f0adf8aa7" (UID: "82976de6-070a-4904-b91c-6f6f0adf8aa7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:59.439727 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.439696    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82976de6-070a-4904-b91c-6f6f0adf8aa7-kube-api-access-vwn5c" pod "82976de6-070a-4904-b91c-6f6f0adf8aa7" (UID: "82976de6-070a-4904-b91c-6f6f0adf8aa7"). InnerVolumeSpecName "kube-api-access-vwn5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:59.440042 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.440019    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-metrics-certs" pod "82976de6-070a-4904-b91c-6f6f0adf8aa7" (UID: "82976de6-070a-4904-b91c-6f6f0adf8aa7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:59.529626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.529541    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82976de6-070a-4904-b91c-6f6f0adf8aa7-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:59.529626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.529571    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:59.529626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.529602    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwn5c\" (UniqueName: \"kubernetes.io/projected/82976de6-070a-4904-b91c-6f6f0adf8aa7-kube-api-access-vwn5c\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:59.529626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:07:59.529613    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82976de6-070a-4904-b91c-6f6f0adf8aa7-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.121475 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.121438    4912 generic.go:356] "Generic (PLEG): container finished" podID="82976de6-070a-4904-b91c-6f6f0adf8aa7" containerID="0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66" exitCode=0
Nov 02 02:08:00.121959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.121491    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"
Nov 02 02:08:00.121959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.121529    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk" event={"ID":"82976de6-070a-4904-b91c-6f6f0adf8aa7","Type":"ContainerDied","Data":"0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66"}
Nov 02 02:08:00.121959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.121605    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk" event={"ID":"82976de6-070a-4904-b91c-6f6f0adf8aa7","Type":"ContainerDied","Data":"76da9a5c3ecee3c6f743fe133162e3ce4783d9acd1d19e38b9af8984cb0eb027"}
Nov 02 02:08:00.121959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.121639    4912 scope.go:122] "RemoveContainer" containerID="0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66"
Nov 02 02:08:00.127289 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.127257    4912 generic.go:356] "Generic (PLEG): container finished" podID="9e7613f4-bddc-450d-aa39-2f88e37759e5" containerID="080c32ac169b20aaf4de877148517c6c5342699c42aaf343b382894355a7f06f" exitCode=0
Nov 02 02:08:00.127484 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.127331    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8" event={"ID":"9e7613f4-bddc-450d-aa39-2f88e37759e5","Type":"ContainerDied","Data":"080c32ac169b20aaf4de877148517c6c5342699c42aaf343b382894355a7f06f"}
Nov 02 02:08:00.137900 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.137877    4912 scope.go:122] "RemoveContainer" containerID="0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66"
Nov 02 02:08:00.138328 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:08:00.138299    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66\": container with ID starting with 0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66 not found: ID does not exist" containerID="0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66"
Nov 02 02:08:00.138453 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.138335    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66"} err="failed to get container status \"0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66\": rpc error: code = NotFound desc = could not find container \"0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66\": container with ID starting with 0c0f9d139f4a917a3090cd7d431aab3bb1ca3bd8211b820f7a97c09c5032ad66 not found: ID does not exist"
Nov 02 02:08:00.178321 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:08:00.178287    4912 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82976de6_070a_4904_b91c_6f6f0adf8aa7.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 02:08:00.293257 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.293212    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"]
Nov 02 02:08:00.299140 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.299113    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-606ca0-7bfd7ff859-tjjgk"]
Nov 02 02:08:00.321348 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.321316    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:00.691649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:00.691606    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:01.105441 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.105354    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:01.230019 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.229984    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82976de6-070a-4904-b91c-6f6f0adf8aa7" path="/var/lib/kubelet/pods/82976de6-070a-4904-b91c-6f6f0adf8aa7/volumes"
Nov 02 02:08:01.414487 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.414460    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"]
Nov 02 02:08:01.426902 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.426856    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-606ca0-gpv6q"]
Nov 02 02:08:01.446076 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.444732    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:08:01.588887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.588856    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7613f4-bddc-450d-aa39-2f88e37759e5-kube-api-access-f59m5\" (UniqueName: \"kubernetes.io/projected/9e7613f4-bddc-450d-aa39-2f88e37759e5-kube-api-access-f59m5\") pod \"9e7613f4-bddc-450d-aa39-2f88e37759e5\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") "
Nov 02 02:08:01.589115 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.589048    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-openshift-ca\") pod \"9e7613f4-bddc-450d-aa39-2f88e37759e5\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") "
Nov 02 02:08:01.589183 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.589152    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-xks-ca\") pod \"9e7613f4-bddc-450d-aa39-2f88e37759e5\" (UID: \"9e7613f4-bddc-450d-aa39-2f88e37759e5\") "
Nov 02 02:08:01.590014 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.589978    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-openshift-ca" pod "9e7613f4-bddc-450d-aa39-2f88e37759e5" (UID: "9e7613f4-bddc-450d-aa39-2f88e37759e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:01.592624 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.592593    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-xks-ca" pod "9e7613f4-bddc-450d-aa39-2f88e37759e5" (UID: "9e7613f4-bddc-450d-aa39-2f88e37759e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:01.592755 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.592622    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7613f4-bddc-450d-aa39-2f88e37759e5-kube-api-access-f59m5" pod "9e7613f4-bddc-450d-aa39-2f88e37759e5" (UID: "9e7613f4-bddc-450d-aa39-2f88e37759e5"). InnerVolumeSpecName "kube-api-access-f59m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:01.691255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.691151    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59m5\" (UniqueName: \"kubernetes.io/projected/9e7613f4-bddc-450d-aa39-2f88e37759e5-kube-api-access-f59m5\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:01.691255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.691191    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:01.691255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.691202    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7613f4-bddc-450d-aa39-2f88e37759e5-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:01.792942 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:01.792902    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:02.152908 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.152876    4912 generic.go:356] "Generic (PLEG): container finished" podID="363846bf-0e6d-466e-b458-97597f12d480" containerID="58cdae4d2708055b31e394ca340212be792c95420df3eebe073f630f460a2164" exitCode=0
Nov 02 02:08:02.153154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.152971    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" event={"ID":"363846bf-0e6d-466e-b458-97597f12d480","Type":"ContainerDied","Data":"58cdae4d2708055b31e394ca340212be792c95420df3eebe073f630f460a2164"}
Nov 02 02:08:02.154757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.154731    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"
Nov 02 02:08:02.154893 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.154731    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8" event={"ID":"9e7613f4-bddc-450d-aa39-2f88e37759e5","Type":"ContainerDied","Data":"6105b3dad06d6ad151b6e7369b4058c6b2ceac892b516c7e7a8a739d6ab9e000"}
Nov 02 02:08:02.154961 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.154900    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6105b3dad06d6ad151b6e7369b4058c6b2ceac892b516c7e7a8a739d6ab9e000"
Nov 02 02:08:02.305964 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.305877    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:02.350532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.350507    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:08:02.400748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.400718    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363846bf-0e6d-466e-b458-97597f12d480-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363846bf-0e6d-466e-b458-97597f12d480-istiod-ca-cert\") pod \"363846bf-0e6d-466e-b458-97597f12d480\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") "
Nov 02 02:08:02.400928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.400752    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/363846bf-0e6d-466e-b458-97597f12d480-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363846bf-0e6d-466e-b458-97597f12d480-istio-podinfo\") pod \"363846bf-0e6d-466e-b458-97597f12d480\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") "
Nov 02 02:08:02.400928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.400823    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-certs\") pod \"363846bf-0e6d-466e-b458-97597f12d480\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") "
Nov 02 02:08:02.400928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.400882    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-istio-token\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-istio-token\") pod \"363846bf-0e6d-466e-b458-97597f12d480\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") "
Nov 02 02:08:02.401139 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.400966    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-kube-api-access-kxw4x\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-kube-api-access-kxw4x\") pod \"363846bf-0e6d-466e-b458-97597f12d480\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") "
Nov 02 02:08:02.401139 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.401031    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-socket\") pod \"363846bf-0e6d-466e-b458-97597f12d480\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") "
Nov 02 02:08:02.401328 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.401210    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-certs" pod "363846bf-0e6d-466e-b458-97597f12d480" (UID: "363846bf-0e6d-466e-b458-97597f12d480"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.401328 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.401233    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363846bf-0e6d-466e-b458-97597f12d480-istiod-ca-cert" pod "363846bf-0e6d-466e-b458-97597f12d480" (UID: "363846bf-0e6d-466e-b458-97597f12d480"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:02.401444 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.401419    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-socket" pod "363846bf-0e6d-466e-b458-97597f12d480" (UID: "363846bf-0e6d-466e-b458-97597f12d480"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.401954 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.401931    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363846bf-0e6d-466e-b458-97597f12d480-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.402020 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.401961    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.402020 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.401977    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.403328 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.403303    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/363846bf-0e6d-466e-b458-97597f12d480-istio-podinfo" pod "363846bf-0e6d-466e-b458-97597f12d480" (UID: "363846bf-0e6d-466e-b458-97597f12d480"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:02.403422 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.403365    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-kube-api-access-kxw4x" pod "363846bf-0e6d-466e-b458-97597f12d480" (UID: "363846bf-0e6d-466e-b458-97597f12d480"). InnerVolumeSpecName "kube-api-access-kxw4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:02.403422 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.403391    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-istio-token" pod "363846bf-0e6d-466e-b458-97597f12d480" (UID: "363846bf-0e6d-466e-b458-97597f12d480"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:02.502510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.502476    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-data\") pod \"363846bf-0e6d-466e-b458-97597f12d480\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") "
Nov 02 02:08:02.502719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.502565    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-credential-socket\") pod \"363846bf-0e6d-466e-b458-97597f12d480\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") "
Nov 02 02:08:02.502719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.502600    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-envoy\") pod \"363846bf-0e6d-466e-b458-97597f12d480\" (UID: \"363846bf-0e6d-466e-b458-97597f12d480\") "
Nov 02 02:08:02.503097 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.503050    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-data" pod "363846bf-0e6d-466e-b458-97597f12d480" (UID: "363846bf-0e6d-466e-b458-97597f12d480"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.503195 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.503109    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363846bf-0e6d-466e-b458-97597f12d480-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.503195 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.503131    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.503195 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.503147    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxw4x\" (UniqueName: \"kubernetes.io/projected/363846bf-0e6d-466e-b458-97597f12d480-kube-api-access-kxw4x\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.503368 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.503298    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-credential-socket" pod "363846bf-0e6d-466e-b458-97597f12d480" (UID: "363846bf-0e6d-466e-b458-97597f12d480"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.506010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.505987    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-envoy" pod "363846bf-0e6d-466e-b458-97597f12d480" (UID: "363846bf-0e6d-466e-b458-97597f12d480"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.600720 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.600623    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:02.604203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.604178    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.604203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.604203    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.604343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.604212    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363846bf-0e6d-466e-b458-97597f12d480-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.985255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:02.985212    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:03.168356 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:03.168328    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"
Nov 02 02:08:03.168575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:03.168363    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2" event={"ID":"363846bf-0e6d-466e-b458-97597f12d480","Type":"ContainerDied","Data":"761933e4a2549bc8819dd3fab92b4d01131d8a77aab5ce8857ecf7e0d100cf1d"}
Nov 02 02:08:03.168575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:03.168448    4912 scope.go:122] "RemoveContainer" containerID="58cdae4d2708055b31e394ca340212be792c95420df3eebe073f630f460a2164"
Nov 02 02:08:03.172404 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:03.172327    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"]
Nov 02 02:08:03.178463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:03.178438    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-zn9wv"]
Nov 02 02:08:03.227155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:03.227128    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e90a7da-f4fc-4ebc-994f-c6a22dc023f0" path="/var/lib/kubelet/pods/0e90a7da-f4fc-4ebc-994f-c6a22dc023f0/volumes"
Nov 02 02:08:03.227579 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:03.227564    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd2e80c-04ca-425f-925b-e56edde1023c" path="/var/lib/kubelet/pods/dcd2e80c-04ca-425f-925b-e56edde1023c/volumes"
Nov 02 02:08:03.340026 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:03.339937    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"]
Nov 02 02:08:03.344272 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:03.344247    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fpqc2"]
Nov 02 02:08:05.236234 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:05.236197    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363846bf-0e6d-466e-b458-97597f12d480" path="/var/lib/kubelet/pods/363846bf-0e6d-466e-b458-97597f12d480/volumes"
Nov 02 02:08:05.728765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:05.728732    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk" podUID="2fa22154-c3f8-47d0-8523-109b05e3a993" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:08:07.407167 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:07.407131    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:07.635169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:07.635139    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:07.840377 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:07.840281    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:08.029434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:08.029399    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:08.333716 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:08.333679    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:09.211083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211040    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"]
Nov 02 02:08:09.211746 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211731    4912 cpu_manager.go:414] "containerMap: removing container" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy"
Nov 02 02:08:09.211801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211747    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy"
Nov 02 02:08:09.211801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211760    4912 cpu_manager.go:414] "containerMap: removing container" podUID="9e7613f4-bddc-450d-aa39-2f88e37759e5" containerName="revoke-keys"
Nov 02 02:08:09.211801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211764    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7613f4-bddc-450d-aa39-2f88e37759e5" containerName="revoke-keys"
Nov 02 02:08:09.211801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211773    4912 cpu_manager.go:414] "containerMap: removing container" podUID="82976de6-070a-4904-b91c-6f6f0adf8aa7" containerName="maas-api"
Nov 02 02:08:09.211801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211778    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82976de6-070a-4904-b91c-6f6f0adf8aa7" containerName="maas-api"
Nov 02 02:08:09.211955 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211876    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="363846bf-0e6d-466e-b458-97597f12d480" containerName="istio-proxy"
Nov 02 02:08:09.211955 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211887    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7613f4-bddc-450d-aa39-2f88e37759e5" containerName="revoke-keys"
Nov 02 02:08:09.211955 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.211893    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="82976de6-070a-4904-b91c-6f6f0adf8aa7" containerName="maas-api"
Nov 02 02:08:09.216963 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.216946    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.299771 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.299735    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"]
Nov 02 02:08:09.314490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.314457    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.314739 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.314721    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwsn\" (UniqueName: \"kubernetes.io/projected/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-kube-api-access-mcwsn\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.314810 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.314772    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.316823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.316797    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:08:09.317044 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.316961    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:08:09.416854 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.416734    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwsn\" (UniqueName: \"kubernetes.io/projected/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-kube-api-access-mcwsn\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.417184 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.416900    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.417184 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.417012    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.418402 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.418376    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.420295 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.420269    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.479386 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.479310    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwsn\" (UniqueName: \"kubernetes.io/projected/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-kube-api-access-mcwsn\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.532753 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.532723    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:09.916724 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:08:09.916697    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb47661_6a60_46e3_9ce9_9e4ac1afbee0.slice/crio-a0472521efb515d75e077ef7f8218b3c1056ab8d88fd5c00d1d29ee9308ec597 WatchSource:0}: Error finding container a0472521efb515d75e077ef7f8218b3c1056ab8d88fd5c00d1d29ee9308ec597: Status 404 returned error can't find the container with id a0472521efb515d75e077ef7f8218b3c1056ab8d88fd5c00d1d29ee9308ec597
Nov 02 02:08:09.917393 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:09.917367    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"]
Nov 02 02:08:10.236025 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:10.235991    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j" event={"ID":"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0","Type":"ContainerStarted","Data":"a0472521efb515d75e077ef7f8218b3c1056ab8d88fd5c00d1d29ee9308ec597"}
Nov 02 02:08:10.436220 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:10.436037    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"]
Nov 02 02:08:10.808349 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:10.808255    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:11.032338 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:11.032295    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:11.245725 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:11.245690    4912 generic.go:356] "Generic (PLEG): container finished" podID="3eb47661-6a60-46e3-9ce9-9e4ac1afbee0" containerID="53ffb8f536c83c1a1c37b2b09c5b6632b1d9cff79b744714fa2575051a5104da" exitCode=0
Nov 02 02:08:11.246166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:11.245774    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j" event={"ID":"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0","Type":"ContainerDied","Data":"53ffb8f536c83c1a1c37b2b09c5b6632b1d9cff79b744714fa2575051a5104da"}
Nov 02 02:08:11.289768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:11.289733    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:11.509707 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:11.509616    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:12.159616 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:08:12.159496    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:08:12.441106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.441084    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:12.480903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.480868    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-openshift-ca\") pod \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") "
Nov 02 02:08:12.481156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.480922    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-xks-ca\") pod \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") "
Nov 02 02:08:12.481280 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.481255    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-kube-api-access-mcwsn\" (UniqueName: \"kubernetes.io/projected/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-kube-api-access-mcwsn\") pod \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\" (UID: \"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0\") "
Nov 02 02:08:12.481356 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.481329    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-openshift-ca" pod "3eb47661-6a60-46e3-9ce9-9e4ac1afbee0" (UID: "3eb47661-6a60-46e3-9ce9-9e4ac1afbee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:12.482005 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.481905    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:12.483432 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.483408    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-xks-ca" pod "3eb47661-6a60-46e3-9ce9-9e4ac1afbee0" (UID: "3eb47661-6a60-46e3-9ce9-9e4ac1afbee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:12.483582 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.483562    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-kube-api-access-mcwsn" pod "3eb47661-6a60-46e3-9ce9-9e4ac1afbee0" (UID: "3eb47661-6a60-46e3-9ce9-9e4ac1afbee0"). InnerVolumeSpecName "kube-api-access-mcwsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:12.583708 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.583661    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:12.583708 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.583703    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcwsn\" (UniqueName: \"kubernetes.io/projected/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0-kube-api-access-mcwsn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:12.751710 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.751672    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:12.812013 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.811988    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 02:08:12.894290 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.894253    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa22154-c3f8-47d0-8523-109b05e3a993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa22154-c3f8-47d0-8523-109b05e3a993-plugins-config-volume\") pod \"2fa22154-c3f8-47d0-8523-109b05e3a993\" (UID: \"2fa22154-c3f8-47d0-8523-109b05e3a993\") "
Nov 02 02:08:12.894487 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.894396    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa22154-c3f8-47d0-8523-109b05e3a993-kube-api-access-7tgdc\" (UniqueName: \"kubernetes.io/projected/2fa22154-c3f8-47d0-8523-109b05e3a993-kube-api-access-7tgdc\") pod \"2fa22154-c3f8-47d0-8523-109b05e3a993\" (UID: \"2fa22154-c3f8-47d0-8523-109b05e3a993\") "
Nov 02 02:08:12.894646 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.894620    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa22154-c3f8-47d0-8523-109b05e3a993-plugins-config-volume" pod "2fa22154-c3f8-47d0-8523-109b05e3a993" (UID: "2fa22154-c3f8-47d0-8523-109b05e3a993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:12.894916 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.894895    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa22154-c3f8-47d0-8523-109b05e3a993-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:12.896721 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.896702    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa22154-c3f8-47d0-8523-109b05e3a993-kube-api-access-7tgdc" pod "2fa22154-c3f8-47d0-8523-109b05e3a993" (UID: "2fa22154-c3f8-47d0-8523-109b05e3a993"). InnerVolumeSpecName "kube-api-access-7tgdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:12.995894 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:12.995805    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tgdc\" (UniqueName: \"kubernetes.io/projected/2fa22154-c3f8-47d0-8523-109b05e3a993-kube-api-access-7tgdc\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.268141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.268046    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"
Nov 02 02:08:13.268141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.268092    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j" event={"ID":"3eb47661-6a60-46e3-9ce9-9e4ac1afbee0","Type":"ContainerDied","Data":"a0472521efb515d75e077ef7f8218b3c1056ab8d88fd5c00d1d29ee9308ec597"}
Nov 02 02:08:13.268141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.268139    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0472521efb515d75e077ef7f8218b3c1056ab8d88fd5c00d1d29ee9308ec597"
Nov 02 02:08:13.269587 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.269561    4912 generic.go:356] "Generic (PLEG): container finished" podID="2fa22154-c3f8-47d0-8523-109b05e3a993" containerID="997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c" exitCode=0
Nov 02 02:08:13.269715 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.269611    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"
Nov 02 02:08:13.269715 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.269646    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk" event={"ID":"2fa22154-c3f8-47d0-8523-109b05e3a993","Type":"ContainerDied","Data":"997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c"}
Nov 02 02:08:13.269715 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.269699    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk" event={"ID":"2fa22154-c3f8-47d0-8523-109b05e3a993","Type":"ContainerDied","Data":"7da395eca1435fab67f42cc08caabfe38ed10631cfd10baba58da4f31a29e492"}
Nov 02 02:08:13.269854 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.269725    4912 scope.go:122] "RemoveContainer" containerID="997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c"
Nov 02 02:08:13.281366 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.281348    4912 scope.go:122] "RemoveContainer" containerID="997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c"
Nov 02 02:08:13.281678 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:08:13.281654    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c\": container with ID starting with 997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c not found: ID does not exist" containerID="997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c"
Nov 02 02:08:13.281735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.281689    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c"} err="failed to get container status \"997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c\": rpc error: code = NotFound desc = could not find container \"997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c\": container with ID starting with 997e937a3e7d3604c496b665295b36fcbf9dff4c41b472deb30f12e2f638a50c not found: ID does not exist"
Nov 02 02:08:13.438156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.438116    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"]
Nov 02 02:08:13.442270 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:13.442244    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed0600-569dbdc4cf-dzdgk"]
Nov 02 02:08:15.227207 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:15.227172    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa22154-c3f8-47d0-8523-109b05e3a993" path="/var/lib/kubelet/pods/2fa22154-c3f8-47d0-8523-109b05e3a993/volumes"
Nov 02 02:08:15.437076 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:15.437021    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv" podUID="d24013f9-f96a-4209-a464-18d6d87c22f1" containerName="payload-processing" containerID="cri-o://de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e" gracePeriod=25
Nov 02 02:08:15.732887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:15.732861    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:08:15.834244 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:15.834164    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24013f9-f96a-4209-a464-18d6d87c22f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24013f9-f96a-4209-a464-18d6d87c22f1-plugins-config-volume\") pod \"d24013f9-f96a-4209-a464-18d6d87c22f1\" (UID: \"d24013f9-f96a-4209-a464-18d6d87c22f1\") "
Nov 02 02:08:15.834244 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:15.834203    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24013f9-f96a-4209-a464-18d6d87c22f1-kube-api-access-hdg2c\" (UniqueName: \"kubernetes.io/projected/d24013f9-f96a-4209-a464-18d6d87c22f1-kube-api-access-hdg2c\") pod \"d24013f9-f96a-4209-a464-18d6d87c22f1\" (UID: \"d24013f9-f96a-4209-a464-18d6d87c22f1\") "
Nov 02 02:08:15.834555 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:15.834535    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24013f9-f96a-4209-a464-18d6d87c22f1-plugins-config-volume" pod "d24013f9-f96a-4209-a464-18d6d87c22f1" (UID: "d24013f9-f96a-4209-a464-18d6d87c22f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:15.834912 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:15.834885    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24013f9-f96a-4209-a464-18d6d87c22f1-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:15.836685 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:15.836664    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24013f9-f96a-4209-a464-18d6d87c22f1-kube-api-access-hdg2c" pod "d24013f9-f96a-4209-a464-18d6d87c22f1" (UID: "d24013f9-f96a-4209-a464-18d6d87c22f1"). InnerVolumeSpecName "kube-api-access-hdg2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:15.936170 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:15.936129    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdg2c\" (UniqueName: \"kubernetes.io/projected/d24013f9-f96a-4209-a464-18d6d87c22f1-kube-api-access-hdg2c\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:16.300819 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.300781    4912 generic.go:356] "Generic (PLEG): container finished" podID="d24013f9-f96a-4209-a464-18d6d87c22f1" containerID="de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e" exitCode=0
Nov 02 02:08:16.301313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.300825    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"
Nov 02 02:08:16.301313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.300856    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv" event={"ID":"d24013f9-f96a-4209-a464-18d6d87c22f1","Type":"ContainerDied","Data":"de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e"}
Nov 02 02:08:16.301313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.300909    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv" event={"ID":"d24013f9-f96a-4209-a464-18d6d87c22f1","Type":"ContainerDied","Data":"02537aa81c8bbb7e6cf4c69f8bf560ba87ba809f57d37014bec81c3f3fd16041"}
Nov 02 02:08:16.301313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.300935    4912 scope.go:122] "RemoveContainer" containerID="de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e"
Nov 02 02:08:16.313588 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.313567    4912 scope.go:122] "RemoveContainer" containerID="de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e"
Nov 02 02:08:16.313897 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:08:16.313877    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e\": container with ID starting with de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e not found: ID does not exist" containerID="de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e"
Nov 02 02:08:16.313951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.313906    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e"} err="failed to get container status \"de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e\": rpc error: code = NotFound desc = could not find container \"de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e\": container with ID starting with de1707d3a1bd4d0d415c7e7cf83c9322ec750c45aed0ed44b94d26565c1da46e not found: ID does not exist"
Nov 02 02:08:16.466628 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.466581    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"]
Nov 02 02:08:16.474806 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.474777    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d1405a-86bff6d67c-h49qv"]
Nov 02 02:08:16.594300 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.594214    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:16.785678 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:16.785637    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:17.238858 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:17.238825    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24013f9-f96a-4209-a464-18d6d87c22f1" path="/var/lib/kubelet/pods/d24013f9-f96a-4209-a464-18d6d87c22f1/volumes"
Nov 02 02:08:18.141489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:18.141450    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:18.476971 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:18.476696    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:18.685351 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:18.685309    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:20.476247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:20.476210    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:20.661534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:20.661501    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:23.304202 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:23.304163    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:24.591185 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:24.591139    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:24.781100 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:24.781044    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:27.679137 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.679032    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"]
Nov 02 02:08:27.679987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.679964    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d24013f9-f96a-4209-a464-18d6d87c22f1" containerName="payload-processing"
Nov 02 02:08:27.680141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.679989    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24013f9-f96a-4209-a464-18d6d87c22f1" containerName="payload-processing"
Nov 02 02:08:27.680141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.680024    4912 cpu_manager.go:414] "containerMap: removing container" podUID="2fa22154-c3f8-47d0-8523-109b05e3a993" containerName="payload-pre-processing"
Nov 02 02:08:27.680141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.680034    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa22154-c3f8-47d0-8523-109b05e3a993" containerName="payload-pre-processing"
Nov 02 02:08:27.680141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.680048    4912 cpu_manager.go:414] "containerMap: removing container" podUID="3eb47661-6a60-46e3-9ce9-9e4ac1afbee0" containerName="revoke-keys"
Nov 02 02:08:27.680141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.680076    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb47661-6a60-46e3-9ce9-9e4ac1afbee0" containerName="revoke-keys"
Nov 02 02:08:27.680418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.680242    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa22154-c3f8-47d0-8523-109b05e3a993" containerName="payload-pre-processing"
Nov 02 02:08:27.680418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.680261    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb47661-6a60-46e3-9ce9-9e4ac1afbee0" containerName="revoke-keys"
Nov 02 02:08:27.680418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.680284    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24013f9-f96a-4209-a464-18d6d87c22f1" containerName="payload-processing"
Nov 02 02:08:27.685446 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.685425    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.709125 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.709101    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8g5j\" (UniqueName: \"kubernetes.io/projected/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-kube-api-access-r8g5j\") pod \"maas-api-revoke-sub-4326c5bf-vwbbp\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.709250 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.709144    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4326c5bf-vwbbp\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.709299 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.709285    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4326c5bf-vwbbp\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.736208 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.736178    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"]
Nov 02 02:08:27.740774 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.740753    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:27.741603 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.741579    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"]
Nov 02 02:08:27.781103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.781038    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:08:27.781451 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.781410    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:08:27.802503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.802449    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"]
Nov 02 02:08:27.810928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.810899    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cndf5\" (UniqueName: \"kubernetes.io/projected/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-kube-api-access-cndf5\") pod \"maas-api-revoke-sub-b4417ac7-mjxwh\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:27.811114 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.810952    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4326c5bf-vwbbp\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.811194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.811108    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4417ac7-mjxwh\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:27.811194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.811187    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8g5j\" (UniqueName: \"kubernetes.io/projected/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-kube-api-access-r8g5j\") pod \"maas-api-revoke-sub-4326c5bf-vwbbp\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.811303 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.811239    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4326c5bf-vwbbp\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.811303 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.811273    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4417ac7-mjxwh\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:27.812249 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.812215    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4326c5bf-vwbbp\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.813903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.813877    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4326c5bf-vwbbp\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.872920 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.872880    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8g5j\" (UniqueName: \"kubernetes.io/projected/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-kube-api-access-r8g5j\") pod \"maas-api-revoke-sub-4326c5bf-vwbbp\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:27.912520 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.912488    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4417ac7-mjxwh\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:27.912698 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.912594    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4417ac7-mjxwh\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:27.912769 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.912731    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cndf5\" (UniqueName: \"kubernetes.io/projected/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-kube-api-access-cndf5\") pod \"maas-api-revoke-sub-b4417ac7-mjxwh\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:27.913160 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.913135    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4417ac7-mjxwh\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:27.915417 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.915394    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4417ac7-mjxwh\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:27.973453 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:27.973383    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cndf5\" (UniqueName: \"kubernetes.io/projected/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-kube-api-access-cndf5\") pod \"maas-api-revoke-sub-b4417ac7-mjxwh\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:28.000612 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:28.000585    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:28.054434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:28.054403    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:28.385391 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:28.385360    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"]
Nov 02 02:08:28.385534 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:08:28.385420    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1de6f5e_bbdc_42cb_baf1_ce7f7d795860.slice/crio-9fcb876c52861c81a7f71671cba860fc5244a8e1499cec144cf109e86ee0eec3 WatchSource:0}: Error finding container 9fcb876c52861c81a7f71671cba860fc5244a8e1499cec144cf109e86ee0eec3: Status 404 returned error can't find the container with id 9fcb876c52861c81a7f71671cba860fc5244a8e1499cec144cf109e86ee0eec3
Nov 02 02:08:28.414379 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:28.414354    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp" event={"ID":"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860","Type":"ContainerStarted","Data":"9fcb876c52861c81a7f71671cba860fc5244a8e1499cec144cf109e86ee0eec3"}
Nov 02 02:08:28.495873 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:28.495844    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"]
Nov 02 02:08:28.496041 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:08:28.496017    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67d002a_e8d3_44a7_b91c_802ceb2f68f9.slice/crio-e94ac5a9f9c529debdddb532b7b4059425d61673ba9333378e8f84047c9c7a1d WatchSource:0}: Error finding container e94ac5a9f9c529debdddb532b7b4059425d61673ba9333378e8f84047c9c7a1d: Status 404 returned error can't find the container with id e94ac5a9f9c529debdddb532b7b4059425d61673ba9333378e8f84047c9c7a1d
Nov 02 02:08:28.587626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:28.587521    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:29.426864 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:29.426830    4912 generic.go:356] "Generic (PLEG): container finished" podID="c1de6f5e-bbdc-42cb-baf1-ce7f7d795860" containerID="b9471caca5fe41c4f7f4d06dcd52813e417b0aa111fdaed11589ba791c0158ee" exitCode=0
Nov 02 02:08:29.427313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:29.426907    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp" event={"ID":"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860","Type":"ContainerDied","Data":"b9471caca5fe41c4f7f4d06dcd52813e417b0aa111fdaed11589ba791c0158ee"}
Nov 02 02:08:29.428926 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:29.428900    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh" event={"ID":"f67d002a-e8d3-44a7-b91c-802ceb2f68f9","Type":"ContainerStarted","Data":"b4c11f5fecd6a00e0293ae8931bfeff248d79b7988444625c67b7a5392c5aef3"}
Nov 02 02:08:29.429047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:29.428949    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh" event={"ID":"f67d002a-e8d3-44a7-b91c-802ceb2f68f9","Type":"ContainerStarted","Data":"e94ac5a9f9c529debdddb532b7b4059425d61673ba9333378e8f84047c9c7a1d"}
Nov 02 02:08:29.664446 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:29.664329    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh" podStartSLOduration=2.664314242 podStartE2EDuration="2.664314242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:08:29.661523313 +0000 UTC m=+2202.566988671" watchObservedRunningTime="2026-11-02 02:08:29.664314242 +0000 UTC m=+2202.569779598"
Nov 02 02:08:30.439205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.439173    4912 generic.go:356] "Generic (PLEG): container finished" podID="f67d002a-e8d3-44a7-b91c-802ceb2f68f9" containerID="b4c11f5fecd6a00e0293ae8931bfeff248d79b7988444625c67b7a5392c5aef3" exitCode=0
Nov 02 02:08:30.439630 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.439229    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh" event={"ID":"f67d002a-e8d3-44a7-b91c-802ceb2f68f9","Type":"ContainerDied","Data":"b4c11f5fecd6a00e0293ae8931bfeff248d79b7988444625c67b7a5392c5aef3"}
Nov 02 02:08:30.579216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.579182    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:30.613993 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.613963    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:30.647526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.647475    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-kube-api-access-r8g5j\" (UniqueName: \"kubernetes.io/projected/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-kube-api-access-r8g5j\") pod \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") "
Nov 02 02:08:30.647526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.647525    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-xks-ca\") pod \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") "
Nov 02 02:08:30.647863 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.647839    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-openshift-ca\") pod \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\" (UID: \"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860\") "
Nov 02 02:08:30.648261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.648234    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-openshift-ca" pod "c1de6f5e-bbdc-42cb-baf1-ce7f7d795860" (UID: "c1de6f5e-bbdc-42cb-baf1-ce7f7d795860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:30.648485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.648469    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:30.649786 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.649759    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-xks-ca" pod "c1de6f5e-bbdc-42cb-baf1-ce7f7d795860" (UID: "c1de6f5e-bbdc-42cb-baf1-ce7f7d795860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:30.649951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.649935    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-kube-api-access-r8g5j" pod "c1de6f5e-bbdc-42cb-baf1-ce7f7d795860" (UID: "c1de6f5e-bbdc-42cb-baf1-ce7f7d795860"). InnerVolumeSpecName "kube-api-access-r8g5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:30.749993 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.749894    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8g5j\" (UniqueName: \"kubernetes.io/projected/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-kube-api-access-r8g5j\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:30.749993 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.749937    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:30.774102 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:30.774050    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:31.449707 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.449665    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"
Nov 02 02:08:31.449707 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.449694    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp" event={"ID":"c1de6f5e-bbdc-42cb-baf1-ce7f7d795860","Type":"ContainerDied","Data":"9fcb876c52861c81a7f71671cba860fc5244a8e1499cec144cf109e86ee0eec3"}
Nov 02 02:08:31.450260 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.449752    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fcb876c52861c81a7f71671cba860fc5244a8e1499cec144cf109e86ee0eec3"
Nov 02 02:08:31.617722 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.617699    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:31.662067 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.662016    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-xks-ca\") pod \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") "
Nov 02 02:08:31.662271 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.662147    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-openshift-ca\") pod \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") "
Nov 02 02:08:31.662271 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.662184    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-kube-api-access-cndf5\" (UniqueName: \"kubernetes.io/projected/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-kube-api-access-cndf5\") pod \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\" (UID: \"f67d002a-e8d3-44a7-b91c-802ceb2f68f9\") "
Nov 02 02:08:31.662566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.662540    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-openshift-ca" pod "f67d002a-e8d3-44a7-b91c-802ceb2f68f9" (UID: "f67d002a-e8d3-44a7-b91c-802ceb2f68f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:31.662727 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.662705    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:31.664593 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.664564    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-xks-ca" pod "f67d002a-e8d3-44a7-b91c-802ceb2f68f9" (UID: "f67d002a-e8d3-44a7-b91c-802ceb2f68f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:31.664724 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.664567    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-kube-api-access-cndf5" pod "f67d002a-e8d3-44a7-b91c-802ceb2f68f9" (UID: "f67d002a-e8d3-44a7-b91c-802ceb2f68f9"). InnerVolumeSpecName "kube-api-access-cndf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:31.764782 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.764676    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:31.764782 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:31.764715    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cndf5\" (UniqueName: \"kubernetes.io/projected/f67d002a-e8d3-44a7-b91c-802ceb2f68f9-kube-api-access-cndf5\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:32.460374 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:32.460341    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"
Nov 02 02:08:32.460788 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:32.460346    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh" event={"ID":"f67d002a-e8d3-44a7-b91c-802ceb2f68f9","Type":"ContainerDied","Data":"e94ac5a9f9c529debdddb532b7b4059425d61673ba9333378e8f84047c9c7a1d"}
Nov 02 02:08:32.460788 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:32.460469    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e94ac5a9f9c529debdddb532b7b4059425d61673ba9333378e8f84047c9c7a1d"
Nov 02 02:08:33.373164 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:33.373130    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:38.253287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:38.251637    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:42.140926 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:08:42.140891    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:08:42.368502 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:42.368454    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:43.401588 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.401546    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"]
Nov 02 02:08:43.402117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.401981    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj" podUID="e27ad0c3-b787-40e4-a50f-7442d9af30c9" containerName="maas-api" containerID="cri-o://57aaa11a66a98768db38a9edc99026fa7a8254747273c56ec7b5281cad0ac650" gracePeriod=30
Nov 02 02:08:43.528798 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.528765    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:43.534711 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.534663    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj" podUID="e27ad0c3-b787-40e4-a50f-7442d9af30c9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.174:8443/health\": dial tcp 10.132.0.174:8443: connect: connection refused"
Nov 02 02:08:43.563342 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.563303    4912 generic.go:356] "Generic (PLEG): container finished" podID="e27ad0c3-b787-40e4-a50f-7442d9af30c9" containerID="57aaa11a66a98768db38a9edc99026fa7a8254747273c56ec7b5281cad0ac650" exitCode=0
Nov 02 02:08:43.563612 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.563589    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj" event={"ID":"e27ad0c3-b787-40e4-a50f-7442d9af30c9","Type":"ContainerDied","Data":"57aaa11a66a98768db38a9edc99026fa7a8254747273c56ec7b5281cad0ac650"}
Nov 02 02:08:43.770629 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.770599    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:08:43.874641 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.874609    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27ad0c3-b787-40e4-a50f-7442d9af30c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ad0c3-b787-40e4-a50f-7442d9af30c9-openshift-service-ca\") pod \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") "
Nov 02 02:08:43.874836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.874749    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls\") pod \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") "
Nov 02 02:08:43.874836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.874798    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-metrics-certs\") pod \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") "
Nov 02 02:08:43.874836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.874833    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27ad0c3-b787-40e4-a50f-7442d9af30c9-kube-api-access-hwgtz\" (UniqueName: \"kubernetes.io/projected/e27ad0c3-b787-40e4-a50f-7442d9af30c9-kube-api-access-hwgtz\") pod \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\" (UID: \"e27ad0c3-b787-40e4-a50f-7442d9af30c9\") "
Nov 02 02:08:43.875015 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.874931    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27ad0c3-b787-40e4-a50f-7442d9af30c9-openshift-service-ca" pod "e27ad0c3-b787-40e4-a50f-7442d9af30c9" (UID: "e27ad0c3-b787-40e4-a50f-7442d9af30c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:43.875363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.875335    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ad0c3-b787-40e4-a50f-7442d9af30c9-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:43.877203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.877175    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27ad0c3-b787-40e4-a50f-7442d9af30c9-kube-api-access-hwgtz" pod "e27ad0c3-b787-40e4-a50f-7442d9af30c9" (UID: "e27ad0c3-b787-40e4-a50f-7442d9af30c9"). InnerVolumeSpecName "kube-api-access-hwgtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:43.877203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.877196    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls" pod "e27ad0c3-b787-40e4-a50f-7442d9af30c9" (UID: "e27ad0c3-b787-40e4-a50f-7442d9af30c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:43.877355 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.877260    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-metrics-certs" pod "e27ad0c3-b787-40e4-a50f-7442d9af30c9" (UID: "e27ad0c3-b787-40e4-a50f-7442d9af30c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:43.977254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.977174    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:43.977254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.977203    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27ad0c3-b787-40e4-a50f-7442d9af30c9-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:43.977254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:43.977213    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwgtz\" (UniqueName: \"kubernetes.io/projected/e27ad0c3-b787-40e4-a50f-7442d9af30c9-kube-api-access-hwgtz\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:44.169324 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:44.169285    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:44.461772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:44.461739    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:44.573614 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:44.573586    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"
Nov 02 02:08:44.573814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:44.573619    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj" event={"ID":"e27ad0c3-b787-40e4-a50f-7442d9af30c9","Type":"ContainerDied","Data":"280fd30f541d30f379c3d5f49e370f1a66074c51c3f47bbc06150281317886d5"}
Nov 02 02:08:44.573814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:44.573702    4912 scope.go:122] "RemoveContainer" containerID="57aaa11a66a98768db38a9edc99026fa7a8254747273c56ec7b5281cad0ac650"
Nov 02 02:08:44.739841 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:44.739757    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"]
Nov 02 02:08:44.743716 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:44.743695    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-47eb54-d866fff54-6bdnj"]
Nov 02 02:08:44.774719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:44.774686    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"]
Nov 02 02:08:44.788504 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:44.788480    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e79d7-gslv6"]
Nov 02 02:08:45.116399 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:45.116356    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:45.226027 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:45.225995    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130d495f-9c80-4c13-8577-aaede795f977" path="/var/lib/kubelet/pods/130d495f-9c80-4c13-8577-aaede795f977/volumes"
Nov 02 02:08:45.226563 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:45.226547    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27ad0c3-b787-40e4-a50f-7442d9af30c9" path="/var/lib/kubelet/pods/e27ad0c3-b787-40e4-a50f-7442d9af30c9/volumes"
Nov 02 02:08:45.476355 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:45.476259    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:45.655426 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:45.655394    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:45.832454 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:45.832374    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:46.025689 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:46.025652    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"]
Nov 02 02:08:46.030327 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:46.030299    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-47eb54-6f2qq"]
Nov 02 02:08:46.436169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:46.436136    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:47.227635 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:47.227604    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be329a34-1d02-47c6-87d2-3224449b303c" path="/var/lib/kubelet/pods/be329a34-1d02-47c6-87d2-3224449b303c/volumes"
Nov 02 02:08:49.745943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:49.745908    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"]
Nov 02 02:08:49.746448 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:49.746130    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy" containerID="cri-o://e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0" gracePeriod=30
Nov 02 02:08:50.027269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:50.027179    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:50.519928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:50.519879    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:51.315746 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:51.315710    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:51.496612 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:51.496569    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:51.680456 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:51.680417    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:51.870107 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:51.870077    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:52.056924 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:52.056813    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:52.232921 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:52.232880    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:52.241238 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:52.241213    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"]
Nov 02 02:08:52.292998 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:52.292962    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d1405a-m4wn8"]
Nov 02 02:08:52.597909 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:52.597868    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:52.774848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:52.774804    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:52.978982 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:52.978927    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:53.226677 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.226640    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7613f4-bddc-450d-aa39-2f88e37759e5" path="/var/lib/kubelet/pods/9e7613f4-bddc-450d-aa39-2f88e37759e5/volumes"
Nov 02 02:08:53.425775 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.425737    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"]
Nov 02 02:08:53.426437 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.426418    4912 cpu_manager.go:414] "containerMap: removing container" podUID="e27ad0c3-b787-40e4-a50f-7442d9af30c9" containerName="maas-api"
Nov 02 02:08:53.426496 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.426441    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27ad0c3-b787-40e4-a50f-7442d9af30c9" containerName="maas-api"
Nov 02 02:08:53.426546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.426496    4912 cpu_manager.go:414] "containerMap: removing container" podUID="c1de6f5e-bbdc-42cb-baf1-ce7f7d795860" containerName="revoke-keys"
Nov 02 02:08:53.426546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.426505    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1de6f5e-bbdc-42cb-baf1-ce7f7d795860" containerName="revoke-keys"
Nov 02 02:08:53.426546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.426532    4912 cpu_manager.go:414] "containerMap: removing container" podUID="f67d002a-e8d3-44a7-b91c-802ceb2f68f9" containerName="revoke-keys"
Nov 02 02:08:53.426546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.426540    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67d002a-e8d3-44a7-b91c-802ceb2f68f9" containerName="revoke-keys"
Nov 02 02:08:53.426736 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.426721    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1de6f5e-bbdc-42cb-baf1-ce7f7d795860" containerName="revoke-keys"
Nov 02 02:08:53.426797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.426741    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67d002a-e8d3-44a7-b91c-802ceb2f68f9" containerName="revoke-keys"
Nov 02 02:08:53.426797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.426753    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27ad0c3-b787-40e4-a50f-7442d9af30c9" containerName="maas-api"
Nov 02 02:08:53.432021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.432003    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.452265 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.452237    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6449\" (UniqueName: \"kubernetes.io/projected/17299572-06c7-4ac0-9bde-2cc8ac874cb5-kube-api-access-f6449\") pod \"maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.452367 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.452350    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.452418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.452376    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.489036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.488997    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"]
Nov 02 02:08:53.531605 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.531573    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:08:53.531864 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.531665    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:08:53.553349 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.553318    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6449\" (UniqueName: \"kubernetes.io/projected/17299572-06c7-4ac0-9bde-2cc8ac874cb5-kube-api-access-f6449\") pod \"maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.553549 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.553398    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.553549 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.553420    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.554198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.554175    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.555981 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.555958    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.633489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.633456    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6449\" (UniqueName: \"kubernetes.io/projected/17299572-06c7-4ac0-9bde-2cc8ac874cb5-kube-api-access-f6449\") pod \"maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.746358 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.746268    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:53.849623 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:53.849001    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:54.085168 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:54.081384    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:54.195815 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:54.195783    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"]
Nov 02 02:08:54.672049 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:54.671944    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8" event={"ID":"17299572-06c7-4ac0-9bde-2cc8ac874cb5","Type":"ContainerStarted","Data":"14a85277dbb4db219d6f80c7f3e31f27f7d354866ac8298f4ee64e3f89b8ad40"}
Nov 02 02:08:54.672049 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:54.672011    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8" event={"ID":"17299572-06c7-4ac0-9bde-2cc8ac874cb5","Type":"ContainerStarted","Data":"0bf8718f48ad8389ea71e8c2fde8fd6817bc8f4485d650f8b9540dc8e5efef3b"}
Nov 02 02:08:54.796165 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:54.796108    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8" podStartSLOduration=1.796087794 podStartE2EDuration="1.796087794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:08:54.79188503 +0000 UTC m=+2227.697350388" watchObservedRunningTime="2026-11-02 02:08:54.796087794 +0000 UTC m=+2227.701553151"
Nov 02 02:08:55.091117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.089412    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 02:08:55.182339 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182304    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d88d562-14a0-4479-97bc-e9dddfcee365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d88d562-14a0-4479-97bc-e9dddfcee365-istiod-ca-cert\") pod \"5d88d562-14a0-4479-97bc-e9dddfcee365\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") "
Nov 02 02:08:55.182542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182362    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-data\") pod \"5d88d562-14a0-4479-97bc-e9dddfcee365\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") "
Nov 02 02:08:55.182542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182394    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-certs\") pod \"5d88d562-14a0-4479-97bc-e9dddfcee365\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") "
Nov 02 02:08:55.182542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182448    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-credential-socket\") pod \"5d88d562-14a0-4479-97bc-e9dddfcee365\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") "
Nov 02 02:08:55.182542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182485    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-kube-api-access-4888s\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-kube-api-access-4888s\") pod \"5d88d562-14a0-4479-97bc-e9dddfcee365\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") "
Nov 02 02:08:55.182542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182521    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-token\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-token\") pod \"5d88d562-14a0-4479-97bc-e9dddfcee365\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") "
Nov 02 02:08:55.182822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182658    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-data" pod "5d88d562-14a0-4479-97bc-e9dddfcee365" (UID: "5d88d562-14a0-4479-97bc-e9dddfcee365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:55.182822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182673    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-podinfo\") pod \"5d88d562-14a0-4479-97bc-e9dddfcee365\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") "
Nov 02 02:08:55.182822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182728    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-socket\") pod \"5d88d562-14a0-4479-97bc-e9dddfcee365\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") "
Nov 02 02:08:55.182822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182754    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d88d562-14a0-4479-97bc-e9dddfcee365-istiod-ca-cert" pod "5d88d562-14a0-4479-97bc-e9dddfcee365" (UID: "5d88d562-14a0-4479-97bc-e9dddfcee365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:55.182822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.182784    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-envoy\") pod \"5d88d562-14a0-4479-97bc-e9dddfcee365\" (UID: \"5d88d562-14a0-4479-97bc-e9dddfcee365\") "
Nov 02 02:08:55.183764 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.183018    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-socket" pod "5d88d562-14a0-4479-97bc-e9dddfcee365" (UID: "5d88d562-14a0-4479-97bc-e9dddfcee365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:55.183764 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.183086    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-certs" pod "5d88d562-14a0-4479-97bc-e9dddfcee365" (UID: "5d88d562-14a0-4479-97bc-e9dddfcee365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:55.183764 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.183135    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-credential-socket" pod "5d88d562-14a0-4479-97bc-e9dddfcee365" (UID: "5d88d562-14a0-4479-97bc-e9dddfcee365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:55.184469 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.184439    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:55.184469 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.184467    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:55.184680 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.184483    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d88d562-14a0-4479-97bc-e9dddfcee365-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:55.184680 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.184497    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:55.184680 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.184510    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:55.185439 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.185401    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-kube-api-access-4888s" pod "5d88d562-14a0-4479-97bc-e9dddfcee365" (UID: "5d88d562-14a0-4479-97bc-e9dddfcee365"). InnerVolumeSpecName "kube-api-access-4888s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:55.185689 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.185653    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-podinfo" pod "5d88d562-14a0-4479-97bc-e9dddfcee365" (UID: "5d88d562-14a0-4479-97bc-e9dddfcee365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:55.185689 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.185657    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-token" pod "5d88d562-14a0-4479-97bc-e9dddfcee365" (UID: "5d88d562-14a0-4479-97bc-e9dddfcee365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:55.186006 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.185984    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-envoy" pod "5d88d562-14a0-4479-97bc-e9dddfcee365" (UID: "5d88d562-14a0-4479-97bc-e9dddfcee365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:55.222395 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.222357    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:55.286529 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.286499    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4888s\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-kube-api-access-4888s\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:55.286529 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.286528    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:55.286768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.286538    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:55.286768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.286547    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d88d562-14a0-4479-97bc-e9dddfcee365-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:55.451091 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.451021    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:55.684359 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.684323    4912 generic.go:356] "Generic (PLEG): container finished" podID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerID="e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0" exitCode=0
Nov 02 02:08:55.684851 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.684388    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"
Nov 02 02:08:55.684851 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.684430    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" event={"ID":"5d88d562-14a0-4479-97bc-e9dddfcee365","Type":"ContainerDied","Data":"e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0"}
Nov 02 02:08:55.684851 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.684490    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb" event={"ID":"5d88d562-14a0-4479-97bc-e9dddfcee365","Type":"ContainerDied","Data":"c07ab43c12af6519b477844e0382331f668ecbb0b40aeea4c2c0231ba8665c69"}
Nov 02 02:08:55.684851 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.684526    4912 scope.go:122] "RemoveContainer" containerID="e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0"
Nov 02 02:08:55.687366 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.687344    4912 generic.go:356] "Generic (PLEG): container finished" podID="17299572-06c7-4ac0-9bde-2cc8ac874cb5" containerID="14a85277dbb4db219d6f80c7f3e31f27f7d354866ac8298f4ee64e3f89b8ad40" exitCode=0
Nov 02 02:08:55.687575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.687528    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8" event={"ID":"17299572-06c7-4ac0-9bde-2cc8ac874cb5","Type":"ContainerDied","Data":"14a85277dbb4db219d6f80c7f3e31f27f7d354866ac8298f4ee64e3f89b8ad40"}
Nov 02 02:08:55.700358 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.700338    4912 scope.go:122] "RemoveContainer" containerID="e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0"
Nov 02 02:08:55.700630 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:08:55.700611    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0\": container with ID starting with e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0 not found: ID does not exist" containerID="e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0"
Nov 02 02:08:55.700685 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.700640    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0"} err="failed to get container status \"e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0\": rpc error: code = NotFound desc = could not find container \"e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0\": container with ID starting with e8d41c79471ced4b129465c07a7fd521164c09a4479ddb1fb0df98af2dc242d0 not found: ID does not exist"
Nov 02 02:08:55.905633 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.905599    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"]
Nov 02 02:08:55.955226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:55.955144    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xmtdb"]
Nov 02 02:08:56.657496 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:56.657460    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:56.973677 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:56.973595    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:57.012019 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.011990    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17299572-06c7-4ac0-9bde-2cc8ac874cb5-kube-api-access-f6449\" (UniqueName: \"kubernetes.io/projected/17299572-06c7-4ac0-9bde-2cc8ac874cb5-kube-api-access-f6449\") pod \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") "
Nov 02 02:08:57.012228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.012040    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-openshift-ca\") pod \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") "
Nov 02 02:08:57.012228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.012090    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-xks-ca\") pod \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\" (UID: \"17299572-06c7-4ac0-9bde-2cc8ac874cb5\") "
Nov 02 02:08:57.012398 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.012376    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-openshift-ca" pod "17299572-06c7-4ac0-9bde-2cc8ac874cb5" (UID: "17299572-06c7-4ac0-9bde-2cc8ac874cb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:57.012521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.012501    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.014377 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.014355    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-xks-ca" pod "17299572-06c7-4ac0-9bde-2cc8ac874cb5" (UID: "17299572-06c7-4ac0-9bde-2cc8ac874cb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:57.014491 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.014474    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17299572-06c7-4ac0-9bde-2cc8ac874cb5-kube-api-access-f6449" pod "17299572-06c7-4ac0-9bde-2cc8ac874cb5" (UID: "17299572-06c7-4ac0-9bde-2cc8ac874cb5"). InnerVolumeSpecName "kube-api-access-f6449". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:57.113943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.113899    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6449\" (UniqueName: \"kubernetes.io/projected/17299572-06c7-4ac0-9bde-2cc8ac874cb5-kube-api-access-f6449\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.114263 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.114237    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17299572-06c7-4ac0-9bde-2cc8ac874cb5-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.161445 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.161410    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"]
Nov 02 02:08:57.227931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.227852    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" path="/var/lib/kubelet/pods/5d88d562-14a0-4479-97bc-e9dddfcee365/volumes"
Nov 02 02:08:57.291170 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.291134    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:57.709244 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.709206    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"
Nov 02 02:08:57.709441 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.709247    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8" event={"ID":"17299572-06c7-4ac0-9bde-2cc8ac874cb5","Type":"ContainerDied","Data":"0bf8718f48ad8389ea71e8c2fde8fd6817bc8f4485d650f8b9540dc8e5efef3b"}
Nov 02 02:08:57.709441 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:57.709329    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf8718f48ad8389ea71e8c2fde8fd6817bc8f4485d650f8b9540dc8e5efef3b"
Nov 02 02:08:58.294082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:58.293594    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:58.546741 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:58.546653    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:58.730956 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:58.730922    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:58.858785 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:58.858700    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"]
Nov 02 02:08:58.911322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:58.911148    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-11f26c-m862j"]
Nov 02 02:08:59.122507 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:59.122465    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:59.226901 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:59.226867    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb47661-6a60-46e3-9ce9-9e4ac1afbee0" path="/var/lib/kubelet/pods/3eb47661-6a60-46e3-9ce9-9e4ac1afbee0/volumes"
Nov 02 02:08:59.388139 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:59.365982    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:08:59.551555 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:08:59.551521    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:00.266891 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:00.266860    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:00.329047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:00.329014    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"]
Nov 02 02:09:00.329274 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:00.329251    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" containerID="cri-o://fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a" gracePeriod=30
Nov 02 02:09:01.214990 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:01.214959    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:01.633974 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:01.633937    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:01.648578 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:01.648552    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"]
Nov 02 02:09:01.648770 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:01.648751    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy" containerID="cri-o://32f6ec00a31e69b8100fef64ac75517200e84d84b21432350081410cc5e05efd" gracePeriod=30
Nov 02 02:09:02.050181 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.050087    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"]
Nov 02 02:09:02.050350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.050311    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy" containerID="cri-o://29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec" gracePeriod=30
Nov 02 02:09:02.300027 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.299995    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"]
Nov 02 02:09:02.300822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.300778    4912 cpu_manager.go:414] "containerMap: removing container" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy"
Nov 02 02:09:02.300822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.300803    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy"
Nov 02 02:09:02.300905 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.300858    4912 cpu_manager.go:414] "containerMap: removing container" podUID="17299572-06c7-4ac0-9bde-2cc8ac874cb5" containerName="revoke-keys"
Nov 02 02:09:02.300905 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.300866    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17299572-06c7-4ac0-9bde-2cc8ac874cb5" containerName="revoke-keys"
Nov 02 02:09:02.301020 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.301008    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d88d562-14a0-4479-97bc-e9dddfcee365" containerName="istio-proxy"
Nov 02 02:09:02.301091 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.301032    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="17299572-06c7-4ac0-9bde-2cc8ac874cb5" containerName="revoke-keys"
Nov 02 02:09:02.305336 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.305318    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.361362 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.361329    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:09:02.361531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.361453    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:09:02.362313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.362289    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"]
Nov 02 02:09:02.425139 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.425109    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1f294fb-f6vrr\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.425311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.425220    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg267\" (UniqueName: \"kubernetes.io/projected/d3923e7a-6361-4f74-ba31-517fb9cfc281-kube-api-access-hg267\") pod \"maas-api-revoke-sub-e1f294fb-f6vrr\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.425457 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.425439    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1f294fb-f6vrr\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.488975 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.488937    4912 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:09:02.489183 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.488999    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:09:02.526793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.526763    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1f294fb-f6vrr\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.526945 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.526868    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1f294fb-f6vrr\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.526945 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.526903    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg267\" (UniqueName: \"kubernetes.io/projected/d3923e7a-6361-4f74-ba31-517fb9cfc281-kube-api-access-hg267\") pod \"maas-api-revoke-sub-e1f294fb-f6vrr\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.527642 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.527609    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1f294fb-f6vrr\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.529519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.529493    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1f294fb-f6vrr\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.588004 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.587936    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg267\" (UniqueName: \"kubernetes.io/projected/d3923e7a-6361-4f74-ba31-517fb9cfc281-kube-api-access-hg267\") pod \"maas-api-revoke-sub-e1f294fb-f6vrr\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.618721 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.618694    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:02.851389 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.851308    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:09:02.851389 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.851374    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:09:02.911284 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.911247    4912 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:09:02.911459 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:02.911321    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:09:03.028083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:03.024193    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"]
Nov 02 02:09:03.035653 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:03.035609    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3923e7a_6361_4f74_ba31_517fb9cfc281.slice/crio-a3cb92057585c2b27562d233ecba14bff0a42eac2a755de2e8efd823bcf9015e WatchSource:0}: Error finding container a3cb92057585c2b27562d233ecba14bff0a42eac2a755de2e8efd823bcf9015e: Status 404 returned error can't find the container with id a3cb92057585c2b27562d233ecba14bff0a42eac2a755de2e8efd823bcf9015e
Nov 02 02:09:03.742952 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:03.742912    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:03.794720 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:03.794650    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr" event={"ID":"d3923e7a-6361-4f74-ba31-517fb9cfc281","Type":"ContainerStarted","Data":"b382b7f3c9b27ce65e5c0a6d747efdfa0c2e91ce8726e978bf8da3f5f8c9738c"}
Nov 02 02:09:03.794904 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:03.794780    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr" event={"ID":"d3923e7a-6361-4f74-ba31-517fb9cfc281","Type":"ContainerStarted","Data":"a3cb92057585c2b27562d233ecba14bff0a42eac2a755de2e8efd823bcf9015e"}
Nov 02 02:09:03.913317 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:03.913269    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr" podStartSLOduration=1.9132535499999999 podStartE2EDuration="1.91325355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:03.910292994 +0000 UTC m=+2236.815758353" watchObservedRunningTime="2026-11-02 02:09:03.91325355 +0000 UTC m=+2236.818718956"
Nov 02 02:09:04.811980 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:04.811933    4912 generic.go:356] "Generic (PLEG): container finished" podID="d3923e7a-6361-4f74-ba31-517fb9cfc281" containerID="b382b7f3c9b27ce65e5c0a6d747efdfa0c2e91ce8726e978bf8da3f5f8c9738c" exitCode=0
Nov 02 02:09:04.812526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:04.811994    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr" event={"ID":"d3923e7a-6361-4f74-ba31-517fb9cfc281","Type":"ContainerDied","Data":"b382b7f3c9b27ce65e5c0a6d747efdfa0c2e91ce8726e978bf8da3f5f8c9738c"}
Nov 02 02:09:05.506109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.506048    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"]
Nov 02 02:09:05.600566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.600519    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:05.611097 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.611045    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"]
Nov 02 02:09:05.629075 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.629042    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 02:09:05.776775 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.776742    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-podinfo\") pod \"7a7d7dab-52b3-4786-b338-0920e1f86129\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") "
Nov 02 02:09:05.776981 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.776803    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-socket\") pod \"7a7d7dab-52b3-4786-b338-0920e1f86129\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") "
Nov 02 02:09:05.776981 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.776834    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-certs\") pod \"7a7d7dab-52b3-4786-b338-0920e1f86129\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") "
Nov 02 02:09:05.776981 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.776856    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-envoy\") pod \"7a7d7dab-52b3-4786-b338-0920e1f86129\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") "
Nov 02 02:09:05.776981 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.776903    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-kube-api-access-sfjdc\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-kube-api-access-sfjdc\") pod \"7a7d7dab-52b3-4786-b338-0920e1f86129\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") "
Nov 02 02:09:05.776981 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.776933    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-data\") pod \"7a7d7dab-52b3-4786-b338-0920e1f86129\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") "
Nov 02 02:09:05.776981 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.776967    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-token\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-token\") pod \"7a7d7dab-52b3-4786-b338-0920e1f86129\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") "
Nov 02 02:09:05.777340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.777038    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7d7dab-52b3-4786-b338-0920e1f86129-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7d7dab-52b3-4786-b338-0920e1f86129-istiod-ca-cert\") pod \"7a7d7dab-52b3-4786-b338-0920e1f86129\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") "
Nov 02 02:09:05.777340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.777234    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-credential-socket\") pod \"7a7d7dab-52b3-4786-b338-0920e1f86129\" (UID: \"7a7d7dab-52b3-4786-b338-0920e1f86129\") "
Nov 02 02:09:05.778281 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.777436    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-certs" pod "7a7d7dab-52b3-4786-b338-0920e1f86129" (UID: "7a7d7dab-52b3-4786-b338-0920e1f86129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:05.778281 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.777614    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-data" pod "7a7d7dab-52b3-4786-b338-0920e1f86129" (UID: "7a7d7dab-52b3-4786-b338-0920e1f86129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:05.778281 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.777855    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.778281 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.777872    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.778281 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.777917    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-socket" pod "7a7d7dab-52b3-4786-b338-0920e1f86129" (UID: "7a7d7dab-52b3-4786-b338-0920e1f86129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:05.778281 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.778172    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-credential-socket" pod "7a7d7dab-52b3-4786-b338-0920e1f86129" (UID: "7a7d7dab-52b3-4786-b338-0920e1f86129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:05.778688 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.778637    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7d7dab-52b3-4786-b338-0920e1f86129-istiod-ca-cert" pod "7a7d7dab-52b3-4786-b338-0920e1f86129" (UID: "7a7d7dab-52b3-4786-b338-0920e1f86129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:05.789147 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.789118    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-envoy" pod "7a7d7dab-52b3-4786-b338-0920e1f86129" (UID: "7a7d7dab-52b3-4786-b338-0920e1f86129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:05.789284 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.789145    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-token" pod "7a7d7dab-52b3-4786-b338-0920e1f86129" (UID: "7a7d7dab-52b3-4786-b338-0920e1f86129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:05.791171 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.791115    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-podinfo" pod "7a7d7dab-52b3-4786-b338-0920e1f86129" (UID: "7a7d7dab-52b3-4786-b338-0920e1f86129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:09:05.800710 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.800683    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-kube-api-access-sfjdc" pod "7a7d7dab-52b3-4786-b338-0920e1f86129" (UID: "7a7d7dab-52b3-4786-b338-0920e1f86129"). InnerVolumeSpecName "kube-api-access-sfjdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:05.830078 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.826305    4912 generic.go:356] "Generic (PLEG): container finished" podID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerID="fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a" exitCode=0
Nov 02 02:09:05.830078 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.826518    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"
Nov 02 02:09:05.830078 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.827432    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" event={"ID":"7a7d7dab-52b3-4786-b338-0920e1f86129","Type":"ContainerDied","Data":"fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a"}
Nov 02 02:09:05.830078 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.827521    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn" event={"ID":"7a7d7dab-52b3-4786-b338-0920e1f86129","Type":"ContainerDied","Data":"f1ffac92410598f2d2fc46a367c1cabe2de32fb0a7cd2c6126c4dad75e708aa4"}
Nov 02 02:09:05.830078 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.827555    4912 scope.go:122] "RemoveContainer" containerID="fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a"
Nov 02 02:09:05.864163 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.864135    4912 scope.go:122] "RemoveContainer" containerID="fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a"
Nov 02 02:09:05.864606 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:05.864567    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a\": container with ID starting with fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a not found: ID does not exist" containerID="fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a"
Nov 02 02:09:05.864730 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.864608    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a"} err="failed to get container status \"fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a\": rpc error: code = NotFound desc = could not find container \"fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a\": container with ID starting with fcba4edb11c0fdb877736173e1472a33f8f36379af577b50f2a5607998bc6d8a not found: ID does not exist"
Nov 02 02:09:05.879833 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.879801    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.879833 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.879836    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.880009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.879851    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.880009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.879865    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.880009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.879878    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjdc\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-kube-api-access-sfjdc\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.880009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.879893    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a7d7dab-52b3-4786-b338-0920e1f86129-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.880009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.879906    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7d7dab-52b3-4786-b338-0920e1f86129-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.888361 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:05.888332    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:06.009469 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.009433    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:06.077151 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.077114    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:06.082920 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.082888    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-openshift-ca\") pod \"d3923e7a-6361-4f74-ba31-517fb9cfc281\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") "
Nov 02 02:09:06.083112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.082971    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-xks-ca\") pod \"d3923e7a-6361-4f74-ba31-517fb9cfc281\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") "
Nov 02 02:09:06.083112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.083046    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3923e7a-6361-4f74-ba31-517fb9cfc281-kube-api-access-hg267\" (UniqueName: \"kubernetes.io/projected/d3923e7a-6361-4f74-ba31-517fb9cfc281-kube-api-access-hg267\") pod \"d3923e7a-6361-4f74-ba31-517fb9cfc281\" (UID: \"d3923e7a-6361-4f74-ba31-517fb9cfc281\") "
Nov 02 02:09:06.083379 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.083353    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-openshift-ca" pod "d3923e7a-6361-4f74-ba31-517fb9cfc281" (UID: "d3923e7a-6361-4f74-ba31-517fb9cfc281"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:06.083741 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.083723    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:06.085352 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.085326    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-xks-ca" pod "d3923e7a-6361-4f74-ba31-517fb9cfc281" (UID: "d3923e7a-6361-4f74-ba31-517fb9cfc281"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:06.085500 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.085480    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3923e7a-6361-4f74-ba31-517fb9cfc281-kube-api-access-hg267" pod "d3923e7a-6361-4f74-ba31-517fb9cfc281" (UID: "d3923e7a-6361-4f74-ba31-517fb9cfc281"). InnerVolumeSpecName "kube-api-access-hg267". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:06.115327 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.115301    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"]
Nov 02 02:09:06.121528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.121504    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-rqvfn"]
Nov 02 02:09:06.185528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.185491    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3923e7a-6361-4f74-ba31-517fb9cfc281-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:06.185528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.185530    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg267\" (UniqueName: \"kubernetes.io/projected/d3923e7a-6361-4f74-ba31-517fb9cfc281-kube-api-access-hg267\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:06.266735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.264995    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:06.447613 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.447503    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:06.653383 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.653345    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:06.835363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.835327    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:06.837741 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.837715    4912 generic.go:356] "Generic (PLEG): container finished" podID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerID="32f6ec00a31e69b8100fef64ac75517200e84d84b21432350081410cc5e05efd" exitCode=0
Nov 02 02:09:06.837861 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.837789    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" event={"ID":"8e945abf-cbdf-44ea-ad4f-59aaaa82b264","Type":"ContainerDied","Data":"32f6ec00a31e69b8100fef64ac75517200e84d84b21432350081410cc5e05efd"}
Nov 02 02:09:06.843432 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.841126    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"
Nov 02 02:09:06.843432 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.841149    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr" event={"ID":"d3923e7a-6361-4f74-ba31-517fb9cfc281","Type":"ContainerDied","Data":"a3cb92057585c2b27562d233ecba14bff0a42eac2a755de2e8efd823bcf9015e"}
Nov 02 02:09:06.843432 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.841204    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3cb92057585c2b27562d233ecba14bff0a42eac2a755de2e8efd823bcf9015e"
Nov 02 02:09:06.947052 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:06.947025    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 02:09:07.009583 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.009481    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-certs\") pod \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") "
Nov 02 02:09:07.009805 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.009783    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-certs" pod "8e945abf-cbdf-44ea-ad4f-59aaaa82b264" (UID: "8e945abf-cbdf-44ea-ad4f-59aaaa82b264"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.009897 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.009880    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-credential-socket\") pod \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") "
Nov 02 02:09:07.009975 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.009961    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-kube-api-access-sx8sq\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-kube-api-access-sx8sq\") pod \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") "
Nov 02 02:09:07.010205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.010024    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istiod-ca-cert\") pod \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") "
Nov 02 02:09:07.010205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.010078    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-data\") pod \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") "
Nov 02 02:09:07.010205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.010129    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-envoy\") pod \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") "
Nov 02 02:09:07.010458 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.010218    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-token\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-token\") pod \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") "
Nov 02 02:09:07.010458 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.010261    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-podinfo\") pod \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") "
Nov 02 02:09:07.010574 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.010523    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-socket\") pod \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\" (UID: \"8e945abf-cbdf-44ea-ad4f-59aaaa82b264\") "
Nov 02 02:09:07.011135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.011082    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.048601 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.031956    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-socket" pod "8e945abf-cbdf-44ea-ad4f-59aaaa82b264" (UID: "8e945abf-cbdf-44ea-ad4f-59aaaa82b264"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.048601 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.037176    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-credential-socket" pod "8e945abf-cbdf-44ea-ad4f-59aaaa82b264" (UID: "8e945abf-cbdf-44ea-ad4f-59aaaa82b264"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.048601 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.037361    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istiod-ca-cert" pod "8e945abf-cbdf-44ea-ad4f-59aaaa82b264" (UID: "8e945abf-cbdf-44ea-ad4f-59aaaa82b264"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:07.048601 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.038227    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:07.048601 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.039386    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-podinfo" pod "8e945abf-cbdf-44ea-ad4f-59aaaa82b264" (UID: "8e945abf-cbdf-44ea-ad4f-59aaaa82b264"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:09:07.055660 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.055627    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-data" pod "8e945abf-cbdf-44ea-ad4f-59aaaa82b264" (UID: "8e945abf-cbdf-44ea-ad4f-59aaaa82b264"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.056449 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.056407    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-envoy" pod "8e945abf-cbdf-44ea-ad4f-59aaaa82b264" (UID: "8e945abf-cbdf-44ea-ad4f-59aaaa82b264"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.057526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.057459    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-kube-api-access-sx8sq" pod "8e945abf-cbdf-44ea-ad4f-59aaaa82b264" (UID: "8e945abf-cbdf-44ea-ad4f-59aaaa82b264"). InnerVolumeSpecName "kube-api-access-sx8sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:07.060146 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.058196    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-token" pod "8e945abf-cbdf-44ea-ad4f-59aaaa82b264" (UID: "8e945abf-cbdf-44ea-ad4f-59aaaa82b264"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:07.114181 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.114140    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.114181 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.114170    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.114181 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.114184    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.114463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.114194    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.114463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.114203    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.114463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.114212    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.114463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.114220    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.114463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.114229    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8sq\" (UniqueName: \"kubernetes.io/projected/8e945abf-cbdf-44ea-ad4f-59aaaa82b264-kube-api-access-sx8sq\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.225893 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.225852    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" path="/var/lib/kubelet/pods/7a7d7dab-52b3-4786-b338-0920e1f86129/volumes"
Nov 02 02:09:07.231106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231076    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"]
Nov 02 02:09:07.231656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231639    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy"
Nov 02 02:09:07.231656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231655    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy"
Nov 02 02:09:07.231801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231682    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy"
Nov 02 02:09:07.231801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231687    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy"
Nov 02 02:09:07.231801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231719    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d3923e7a-6361-4f74-ba31-517fb9cfc281" containerName="revoke-keys"
Nov 02 02:09:07.231801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231724    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3923e7a-6361-4f74-ba31-517fb9cfc281" containerName="revoke-keys"
Nov 02 02:09:07.232006 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231806    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" containerName="istio-proxy"
Nov 02 02:09:07.232006 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231817    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7d7dab-52b3-4786-b338-0920e1f86129" containerName="istio-proxy"
Nov 02 02:09:07.232006 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.231828    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3923e7a-6361-4f74-ba31-517fb9cfc281" containerName="revoke-keys"
Nov 02 02:09:07.235463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.235444    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.289596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.288494    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:09:07.289596 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.288576    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:09:07.294514 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.294474    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"]
Nov 02 02:09:07.326740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.317886    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.326740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.317990    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhvc\" (UniqueName: \"kubernetes.io/projected/0c1c0f26-3660-4874-a458-402952ce2612-kube-api-access-wnhvc\") pod \"maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.326740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.318133    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.419732 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.419698    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.419906 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.419773    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhvc\" (UniqueName: \"kubernetes.io/projected/0c1c0f26-3660-4874-a458-402952ce2612-kube-api-access-wnhvc\") pod \"maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.419906 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.419845    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.420407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.420384    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.422332 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.422309    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.480512 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.480479    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhvc\" (UniqueName: \"kubernetes.io/projected/0c1c0f26-3660-4874-a458-402952ce2612-kube-api-access-wnhvc\") pod \"maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.555803 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.555710    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:07.780796 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.780769    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 02:09:07.860007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.859921    4912 generic.go:356] "Generic (PLEG): container finished" podID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerID="29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec" exitCode=0
Nov 02 02:09:07.860007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.859993    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"
Nov 02 02:09:07.860540 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.860010    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" event={"ID":"5e8cfca1-a503-44b7-a3d2-d428da9374aa","Type":"ContainerDied","Data":"29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec"}
Nov 02 02:09:07.860540 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.860144    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh" event={"ID":"5e8cfca1-a503-44b7-a3d2-d428da9374aa","Type":"ContainerDied","Data":"08b9f7822000769dfac8395ac43b7ef977903cf9b69f904091df9534fe95dbba"}
Nov 02 02:09:07.860540 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.860176    4912 scope.go:122] "RemoveContainer" containerID="29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec"
Nov 02 02:09:07.862304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.862283    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"
Nov 02 02:09:07.862413 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.862303    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj" event={"ID":"8e945abf-cbdf-44ea-ad4f-59aaaa82b264","Type":"ContainerDied","Data":"075f7a30c74730ed1a46d11801b8b37eb619d167f24d7ea2fa96843c8f34a83c"}
Nov 02 02:09:07.874795 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.874777    4912 scope.go:122] "RemoveContainer" containerID="29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec"
Nov 02 02:09:07.875126 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:07.875087    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec\": container with ID starting with 29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec not found: ID does not exist" containerID="29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec"
Nov 02 02:09:07.875250 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.875126    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec"} err="failed to get container status \"29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec\": rpc error: code = NotFound desc = could not find container \"29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec\": container with ID starting with 29d54dc7a573b58f5ef0cfd13b8b42f674aa577fe1785ae0816446147b389eec not found: ID does not exist"
Nov 02 02:09:07.875250 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.875149    4912 scope.go:122] "RemoveContainer" containerID="32f6ec00a31e69b8100fef64ac75517200e84d84b21432350081410cc5e05efd"
Nov 02 02:09:07.926176 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926142    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istiod-ca-cert\") pod \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") "
Nov 02 02:09:07.926349 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926189    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-certs\") pod \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") "
Nov 02 02:09:07.926349 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926242    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-data\") pod \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") "
Nov 02 02:09:07.926349 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926292    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-envoy\") pod \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") "
Nov 02 02:09:07.926349 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926318    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-token\") pod \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") "
Nov 02 02:09:07.926572 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926365    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-podinfo\") pod \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") "
Nov 02 02:09:07.926572 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926519    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-socket\") pod \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") "
Nov 02 02:09:07.926723 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926692    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-data" pod "5e8cfca1-a503-44b7-a3d2-d428da9374aa" (UID: "5e8cfca1-a503-44b7-a3d2-d428da9374aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.926802 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926716    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-kube-api-access-dhwrd\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-kube-api-access-dhwrd\") pod \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") "
Nov 02 02:09:07.926802 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926784    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-socket" pod "5e8cfca1-a503-44b7-a3d2-d428da9374aa" (UID: "5e8cfca1-a503-44b7-a3d2-d428da9374aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.926920 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926838    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-credential-socket\") pod \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\" (UID: \"5e8cfca1-a503-44b7-a3d2-d428da9374aa\") "
Nov 02 02:09:07.926988 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.926942    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istiod-ca-cert" pod "5e8cfca1-a503-44b7-a3d2-d428da9374aa" (UID: "5e8cfca1-a503-44b7-a3d2-d428da9374aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:07.927049 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.927017    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-certs" pod "5e8cfca1-a503-44b7-a3d2-d428da9374aa" (UID: "5e8cfca1-a503-44b7-a3d2-d428da9374aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.927275 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.927249    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-credential-socket" pod "5e8cfca1-a503-44b7-a3d2-d428da9374aa" (UID: "5e8cfca1-a503-44b7-a3d2-d428da9374aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.927539 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.927451    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.927539 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.927476    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.927539 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.927490    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.927539 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.927505    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.927539 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.927527    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:07.929262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.929233    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-token" pod "5e8cfca1-a503-44b7-a3d2-d428da9374aa" (UID: "5e8cfca1-a503-44b7-a3d2-d428da9374aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:07.929387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.929367    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-envoy" pod "5e8cfca1-a503-44b7-a3d2-d428da9374aa" (UID: "5e8cfca1-a503-44b7-a3d2-d428da9374aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:07.929529 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.929512    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-kube-api-access-dhwrd" pod "5e8cfca1-a503-44b7-a3d2-d428da9374aa" (UID: "5e8cfca1-a503-44b7-a3d2-d428da9374aa"). InnerVolumeSpecName "kube-api-access-dhwrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:07.929822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.929793    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-podinfo" pod "5e8cfca1-a503-44b7-a3d2-d428da9374aa" (UID: "5e8cfca1-a503-44b7-a3d2-d428da9374aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:09:07.949908 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:07.949878    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"]
Nov 02 02:09:07.988330 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:07.988302    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1c0f26_3660_4874_a458_402952ce2612.slice/crio-cdf212c79df4fe04e24099dd546f3560f2a931fba564a626a35ee5197a181b4a WatchSource:0}: Error finding container cdf212c79df4fe04e24099dd546f3560f2a931fba564a626a35ee5197a181b4a: Status 404 returned error can't find the container with id cdf212c79df4fe04e24099dd546f3560f2a931fba564a626a35ee5197a181b4a
Nov 02 02:09:08.017810 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.017784    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"]
Nov 02 02:09:08.021774 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.021749    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-7bjmj"]
Nov 02 02:09:08.028764 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.028744    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhwrd\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-kube-api-access-dhwrd\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:08.028827 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.028768    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:08.028827 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.028780    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:08.028827 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.028789    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8cfca1-a503-44b7-a3d2-d428da9374aa-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:08.282112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.281241    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"]
Nov 02 02:09:08.282330 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.282298    4912 cpu_manager.go:414] "containerMap: removing container" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy"
Nov 02 02:09:08.282330 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.282318    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy"
Nov 02 02:09:08.282514 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.282500    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" containerName="istio-proxy"
Nov 02 02:09:08.288408 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.288386    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.297049 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.296947    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"]
Nov 02 02:09:08.302595 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.302569    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-clxsh"]
Nov 02 02:09:08.342690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.342662    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"]
Nov 02 02:09:08.433619 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.433524    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gfz\" (UniqueName: \"kubernetes.io/projected/7f637d0b-3394-4af9-8237-8519e63188f4-kube-api-access-28gfz\") pod \"maas-api-revoke-sub-b343b313-kdjk9\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.433790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.433684    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b343b313-kdjk9\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.433790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.433744    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b343b313-kdjk9\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.541181 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.541149    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gfz\" (UniqueName: \"kubernetes.io/projected/7f637d0b-3394-4af9-8237-8519e63188f4-kube-api-access-28gfz\") pod \"maas-api-revoke-sub-b343b313-kdjk9\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.543561 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.543499    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b343b313-kdjk9\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.544188 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.544166    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b343b313-kdjk9\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.545259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.545017    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b343b313-kdjk9\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.549978 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.549952    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b343b313-kdjk9\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.603325 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.603285    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gfz\" (UniqueName: \"kubernetes.io/projected/7f637d0b-3394-4af9-8237-8519e63188f4-kube-api-access-28gfz\") pod \"maas-api-revoke-sub-b343b313-kdjk9\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.607154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.607113    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:08.875443 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.875405    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6" event={"ID":"0c1c0f26-3660-4874-a458-402952ce2612","Type":"ContainerStarted","Data":"e8181a38afe3cc0909876963b00d0a509b24c6d422e400aec4f8de263fa6896c"}
Nov 02 02:09:08.875959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.875474    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6" event={"ID":"0c1c0f26-3660-4874-a458-402952ce2612","Type":"ContainerStarted","Data":"cdf212c79df4fe04e24099dd546f3560f2a931fba564a626a35ee5197a181b4a"}
Nov 02 02:09:08.997520 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:08.997491    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"]
Nov 02 02:09:09.002317 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:09.002271    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6" podStartSLOduration=2.002257731 podStartE2EDuration="2.002257731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:08.993371608 +0000 UTC m=+2241.898836965" watchObservedRunningTime="2026-11-02 02:09:09.002257731 +0000 UTC m=+2241.907723087"
Nov 02 02:09:09.003229 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:09.003205    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f637d0b_3394_4af9_8237_8519e63188f4.slice/crio-7b111d67da1eac1fbf8b28485ff85cf941d1107881c46b9bb1c54622e1f84717 WatchSource:0}: Error finding container 7b111d67da1eac1fbf8b28485ff85cf941d1107881c46b9bb1c54622e1f84717: Status 404 returned error can't find the container with id 7b111d67da1eac1fbf8b28485ff85cf941d1107881c46b9bb1c54622e1f84717
Nov 02 02:09:09.210493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:09.210458    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:09.240547 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:09.240511    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8cfca1-a503-44b7-a3d2-d428da9374aa" path="/var/lib/kubelet/pods/5e8cfca1-a503-44b7-a3d2-d428da9374aa/volumes"
Nov 02 02:09:09.241025 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:09.241008    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e945abf-cbdf-44ea-ad4f-59aaaa82b264" path="/var/lib/kubelet/pods/8e945abf-cbdf-44ea-ad4f-59aaaa82b264/volumes"
Nov 02 02:09:09.889035 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:09.889003    4912 generic.go:356] "Generic (PLEG): container finished" podID="0c1c0f26-3660-4874-a458-402952ce2612" containerID="e8181a38afe3cc0909876963b00d0a509b24c6d422e400aec4f8de263fa6896c" exitCode=0
Nov 02 02:09:09.889559 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:09.889141    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6" event={"ID":"0c1c0f26-3660-4874-a458-402952ce2612","Type":"ContainerDied","Data":"e8181a38afe3cc0909876963b00d0a509b24c6d422e400aec4f8de263fa6896c"}
Nov 02 02:09:09.891787 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:09.891753    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9" event={"ID":"7f637d0b-3394-4af9-8237-8519e63188f4","Type":"ContainerStarted","Data":"5c7a2210c150b8e255c73ab3d9164187b6093333ea988e014fcc5a268bbc41b1"}
Nov 02 02:09:09.891916 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:09.891809    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9" event={"ID":"7f637d0b-3394-4af9-8237-8519e63188f4","Type":"ContainerStarted","Data":"7b111d67da1eac1fbf8b28485ff85cf941d1107881c46b9bb1c54622e1f84717"}
Nov 02 02:09:10.123035 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:10.122985    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9" podStartSLOduration=2.122969936 podStartE2EDuration="2.122969936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:10.120105289 +0000 UTC m=+2243.025570646" watchObservedRunningTime="2026-11-02 02:09:10.122969936 +0000 UTC m=+2243.028435333"
Nov 02 02:09:10.506626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:10.506582    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj" podUID="956e87a3-da69-4672-82ec-772ed56c413a" containerName="payload-processing" containerID="cri-o://03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290" gracePeriod=25
Nov 02 02:09:10.826636 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:10.826547    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx" podUID="633f0926-b18e-4ed5-b916-268160edee9a" containerName="payload-pre-processing" containerID="cri-o://bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85" gracePeriod=25
Nov 02 02:09:10.902203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:10.902171    4912 generic.go:356] "Generic (PLEG): container finished" podID="7f637d0b-3394-4af9-8237-8519e63188f4" containerID="5c7a2210c150b8e255c73ab3d9164187b6093333ea988e014fcc5a268bbc41b1" exitCode=0
Nov 02 02:09:10.902699 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:10.902254    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9" event={"ID":"7f637d0b-3394-4af9-8237-8519e63188f4","Type":"ContainerDied","Data":"5c7a2210c150b8e255c73ab3d9164187b6093333ea988e014fcc5a268bbc41b1"}
Nov 02 02:09:11.110704 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.110675    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:11.199229 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.199189    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:11.283331 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.283299    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1c0f26-3660-4874-a458-402952ce2612-kube-api-access-wnhvc\" (UniqueName: \"kubernetes.io/projected/0c1c0f26-3660-4874-a458-402952ce2612-kube-api-access-wnhvc\") pod \"0c1c0f26-3660-4874-a458-402952ce2612\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") "
Nov 02 02:09:11.283528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.283420    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-openshift-ca\") pod \"0c1c0f26-3660-4874-a458-402952ce2612\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") "
Nov 02 02:09:11.283617 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.283537    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-xks-ca\") pod \"0c1c0f26-3660-4874-a458-402952ce2612\" (UID: \"0c1c0f26-3660-4874-a458-402952ce2612\") "
Nov 02 02:09:11.283824 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.283801    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-openshift-ca" pod "0c1c0f26-3660-4874-a458-402952ce2612" (UID: "0c1c0f26-3660-4874-a458-402952ce2612"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:11.284005 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.283987    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.286404 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.286380    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-xks-ca" pod "0c1c0f26-3660-4874-a458-402952ce2612" (UID: "0c1c0f26-3660-4874-a458-402952ce2612"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:11.286404 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.286390    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1c0f26-3660-4874-a458-402952ce2612-kube-api-access-wnhvc" pod "0c1c0f26-3660-4874-a458-402952ce2612" (UID: "0c1c0f26-3660-4874-a458-402952ce2612"). InnerVolumeSpecName "kube-api-access-wnhvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:11.386010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.385969    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1c0f26-3660-4874-a458-402952ce2612-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.386010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.386006    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhvc\" (UniqueName: \"kubernetes.io/projected/0c1c0f26-3660-4874-a458-402952ce2612-kube-api-access-wnhvc\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.411889 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.411859    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:11.587355 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.587313    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:11.692844 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.692751    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"]
Nov 02 02:09:11.697124 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.697094    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"]
Nov 02 02:09:11.741228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.741193    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"]
Nov 02 02:09:11.827148 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.827117    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:11.912907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.912879    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"
Nov 02 02:09:11.913445 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.912906    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6" event={"ID":"0c1c0f26-3660-4874-a458-402952ce2612","Type":"ContainerDied","Data":"cdf212c79df4fe04e24099dd546f3560f2a931fba564a626a35ee5197a181b4a"}
Nov 02 02:09:11.913445 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:11.912956    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdf212c79df4fe04e24099dd546f3560f2a931fba564a626a35ee5197a181b4a"
Nov 02 02:09:12.072731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.072701    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:12.138474 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:12.138440    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:09:12.191968 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.191930    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" containerID="cri-o://488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a" gracePeriod=45
Nov 02 02:09:12.199764 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.199735    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-xks-ca\") pod \"7f637d0b-3394-4af9-8237-8519e63188f4\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") "
Nov 02 02:09:12.199891 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.199871    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f637d0b-3394-4af9-8237-8519e63188f4-kube-api-access-28gfz\" (UniqueName: \"kubernetes.io/projected/7f637d0b-3394-4af9-8237-8519e63188f4-kube-api-access-28gfz\") pod \"7f637d0b-3394-4af9-8237-8519e63188f4\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") "
Nov 02 02:09:12.200114 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.200097    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-openshift-ca\") pod \"7f637d0b-3394-4af9-8237-8519e63188f4\" (UID: \"7f637d0b-3394-4af9-8237-8519e63188f4\") "
Nov 02 02:09:12.200534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.200509    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-openshift-ca" pod "7f637d0b-3394-4af9-8237-8519e63188f4" (UID: "7f637d0b-3394-4af9-8237-8519e63188f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:12.200662 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.200646    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:12.202095 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.202075    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f637d0b-3394-4af9-8237-8519e63188f4-kube-api-access-28gfz" pod "7f637d0b-3394-4af9-8237-8519e63188f4" (UID: "7f637d0b-3394-4af9-8237-8519e63188f4"). InnerVolumeSpecName "kube-api-access-28gfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:12.202331 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.202303    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-xks-ca" pod "7f637d0b-3394-4af9-8237-8519e63188f4" (UID: "7f637d0b-3394-4af9-8237-8519e63188f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:12.277100 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.277005    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": dial tcp 10.132.0.101:8000: connect: connection refused"
Nov 02 02:09:12.301914 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.301865    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637d0b-3394-4af9-8237-8519e63188f4-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:12.301914 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.301899    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28gfz\" (UniqueName: \"kubernetes.io/projected/7f637d0b-3394-4af9-8237-8519e63188f4-kube-api-access-28gfz\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:12.479586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.479554    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 02:09:12.604566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.604475    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f24ede2a-7f50-43fc-862b-b27ba373340d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ede2a-7f50-43fc-862b-b27ba373340d-tls-certs\") pod \"f24ede2a-7f50-43fc-862b-b27ba373340d\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") "
Nov 02 02:09:12.604740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.604596    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-dshm\") pod \"f24ede2a-7f50-43fc-862b-b27ba373340d\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") "
Nov 02 02:09:12.604740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.604654    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-model-cache\") pod \"f24ede2a-7f50-43fc-862b-b27ba373340d\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") "
Nov 02 02:09:12.604740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.604685    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-tmp-dir\") pod \"f24ede2a-7f50-43fc-862b-b27ba373340d\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") "
Nov 02 02:09:12.604740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.604716    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24ede2a-7f50-43fc-862b-b27ba373340d-kube-api-access-qzslm\" (UniqueName: \"kubernetes.io/projected/f24ede2a-7f50-43fc-862b-b27ba373340d-kube-api-access-qzslm\") pod \"f24ede2a-7f50-43fc-862b-b27ba373340d\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") "
Nov 02 02:09:12.604967 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.604875    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-home\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-home\") pod \"f24ede2a-7f50-43fc-862b-b27ba373340d\" (UID: \"f24ede2a-7f50-43fc-862b-b27ba373340d\") "
Nov 02 02:09:12.605902 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.605874    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-tmp-dir" pod "f24ede2a-7f50-43fc-862b-b27ba373340d" (UID: "f24ede2a-7f50-43fc-862b-b27ba373340d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:12.606929 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.606566    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-model-cache" pod "f24ede2a-7f50-43fc-862b-b27ba373340d" (UID: "f24ede2a-7f50-43fc-862b-b27ba373340d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:12.607130 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.607102    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-home" pod "f24ede2a-7f50-43fc-862b-b27ba373340d" (UID: "f24ede2a-7f50-43fc-862b-b27ba373340d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:12.608638 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.608608    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-dshm" pod "f24ede2a-7f50-43fc-862b-b27ba373340d" (UID: "f24ede2a-7f50-43fc-862b-b27ba373340d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:12.609216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.609187    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f24ede2a-7f50-43fc-862b-b27ba373340d-tls-certs" pod "f24ede2a-7f50-43fc-862b-b27ba373340d" (UID: "f24ede2a-7f50-43fc-862b-b27ba373340d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:12.609500 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.609462    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24ede2a-7f50-43fc-862b-b27ba373340d-kube-api-access-qzslm" pod "f24ede2a-7f50-43fc-862b-b27ba373340d" (UID: "f24ede2a-7f50-43fc-862b-b27ba373340d"). InnerVolumeSpecName "kube-api-access-qzslm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:12.707794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.707044    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:12.707794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.707107    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ede2a-7f50-43fc-862b-b27ba373340d-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:12.707794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.707121    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:12.707794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.707134    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:12.707794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.707147    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ede2a-7f50-43fc-862b-b27ba373340d-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:12.707794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.707161    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzslm\" (UniqueName: \"kubernetes.io/projected/f24ede2a-7f50-43fc-862b-b27ba373340d-kube-api-access-qzslm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:12.922811 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.922780    4912 generic.go:356] "Generic (PLEG): container finished" podID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerID="488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a" exitCode=0
Nov 02 02:09:12.923345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.922839    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"
Nov 02 02:09:12.923345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.922865    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" event={"ID":"f24ede2a-7f50-43fc-862b-b27ba373340d","Type":"ContainerDied","Data":"488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a"}
Nov 02 02:09:12.923345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.922913    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf" event={"ID":"f24ede2a-7f50-43fc-862b-b27ba373340d","Type":"ContainerDied","Data":"d96b328e40aa7fddb78a541d1054bcd72c986eb1e53471554d8bd7ee30067f49"}
Nov 02 02:09:12.923345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.922930    4912 scope.go:122] "RemoveContainer" containerID="488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a"
Nov 02 02:09:12.924503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.924483    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"
Nov 02 02:09:12.924594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.924502    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9" event={"ID":"7f637d0b-3394-4af9-8237-8519e63188f4","Type":"ContainerDied","Data":"7b111d67da1eac1fbf8b28485ff85cf941d1107881c46b9bb1c54622e1f84717"}
Nov 02 02:09:12.924594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.924541    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b111d67da1eac1fbf8b28485ff85cf941d1107881c46b9bb1c54622e1f84717"
Nov 02 02:09:12.935890 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.935875    4912 scope.go:122] "RemoveContainer" containerID="488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a"
Nov 02 02:09:12.936173 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:12.936151    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a\": container with ID starting with 488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a not found: ID does not exist" containerID="488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a"
Nov 02 02:09:12.936222 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.936181    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a"} err="failed to get container status \"488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a\": rpc error: code = NotFound desc = could not find container \"488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a\": container with ID starting with 488c9e47086aaae931f9110e01d2f80c6f647fedb65668aaf2be64f3783bdb8a not found: ID does not exist"
Nov 02 02:09:12.959915 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:12.959879    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:13.102488 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:13.102454    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"]
Nov 02 02:09:13.117838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:13.117804    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-55bc84/distinct-w5-55bc84-kserve-8fddcc5f7-kflkf"]
Nov 02 02:09:13.145756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:13.145709    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:13.243786 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:13.243628    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" path="/var/lib/kubelet/pods/f24ede2a-7f50-43fc-862b-b27ba373340d/volumes"
Nov 02 02:09:14.465711 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:14.465666    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:15.442122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:15.442076    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:15.625676 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:15.625594    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:17.819743 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:17.819695    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj" podUID="956e87a3-da69-4672-82ec-772ed56c413a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:09:17.819743 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:17.819735    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx" podUID="633f0926-b18e-4ed5-b916-268160edee9a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:09:19.035156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.035114    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"]
Nov 02 02:09:19.082469 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.082436    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"]
Nov 02 02:09:19.082661 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.082640    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv" podUID="5c785f0a-71f7-432c-900f-bb08d6b82636" containerName="maas-api" containerID="cri-o://5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb" gracePeriod=30
Nov 02 02:09:19.198742 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.198704    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:19.377690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.377613    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:19.409474 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.409443    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 02:09:19.504017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.503985    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c785f0a-71f7-432c-900f-bb08d6b82636-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c785f0a-71f7-432c-900f-bb08d6b82636-openshift-service-ca\") pod \"5c785f0a-71f7-432c-900f-bb08d6b82636\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") "
Nov 02 02:09:19.504226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.504096    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c785f0a-71f7-432c-900f-bb08d6b82636-kube-api-access-6djv8\" (UniqueName: \"kubernetes.io/projected/5c785f0a-71f7-432c-900f-bb08d6b82636-kube-api-access-6djv8\") pod \"5c785f0a-71f7-432c-900f-bb08d6b82636\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") "
Nov 02 02:09:19.504226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.504162    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls\") pod \"5c785f0a-71f7-432c-900f-bb08d6b82636\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") "
Nov 02 02:09:19.504226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.504192    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-metrics-certs\") pod \"5c785f0a-71f7-432c-900f-bb08d6b82636\" (UID: \"5c785f0a-71f7-432c-900f-bb08d6b82636\") "
Nov 02 02:09:19.504414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.504385    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c785f0a-71f7-432c-900f-bb08d6b82636-openshift-service-ca" pod "5c785f0a-71f7-432c-900f-bb08d6b82636" (UID: "5c785f0a-71f7-432c-900f-bb08d6b82636"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:19.504688 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.504665    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c785f0a-71f7-432c-900f-bb08d6b82636-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:19.506468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.506440    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c785f0a-71f7-432c-900f-bb08d6b82636-kube-api-access-6djv8" pod "5c785f0a-71f7-432c-900f-bb08d6b82636" (UID: "5c785f0a-71f7-432c-900f-bb08d6b82636"). InnerVolumeSpecName "kube-api-access-6djv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:19.506468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.506460    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls" pod "5c785f0a-71f7-432c-900f-bb08d6b82636" (UID: "5c785f0a-71f7-432c-900f-bb08d6b82636"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:19.506611 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.506441    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-metrics-certs" pod "5c785f0a-71f7-432c-900f-bb08d6b82636" (UID: "5c785f0a-71f7-432c-900f-bb08d6b82636"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:19.562626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.562589    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:19.606325 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.606284    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6djv8\" (UniqueName: \"kubernetes.io/projected/5c785f0a-71f7-432c-900f-bb08d6b82636-kube-api-access-6djv8\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:19.606325 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.606319    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:19.606325 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.606332    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c785f0a-71f7-432c-900f-bb08d6b82636-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:19.990894 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.990862    4912 generic.go:356] "Generic (PLEG): container finished" podID="5c785f0a-71f7-432c-900f-bb08d6b82636" containerID="5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb" exitCode=0
Nov 02 02:09:19.991135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.990926    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"
Nov 02 02:09:19.991135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.990950    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv" event={"ID":"5c785f0a-71f7-432c-900f-bb08d6b82636","Type":"ContainerDied","Data":"5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb"}
Nov 02 02:09:19.991135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.991021    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv" event={"ID":"5c785f0a-71f7-432c-900f-bb08d6b82636","Type":"ContainerDied","Data":"cef2abf5ccdaff2a1b9f714aa2e4e91e35b5b48253cd44d9265ee65cd8849664"}
Nov 02 02:09:19.991135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:19.991052    4912 scope.go:122] "RemoveContainer" containerID="5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb"
Nov 02 02:09:20.038754 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:20.017268    4912 scope.go:122] "RemoveContainer" containerID="5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb"
Nov 02 02:09:20.038754 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:20.025132    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb\": container with ID starting with 5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb not found: ID does not exist" containerID="5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb"
Nov 02 02:09:20.038754 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:20.025175    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb"} err="failed to get container status \"5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb\": rpc error: code = NotFound desc = could not find container \"5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb\": container with ID starting with 5e70b63139a4213a2b918a9f655facb251adb77a45c4940d8dc8f304afc732fb not found: ID does not exist"
Nov 02 02:09:20.164468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:20.164438    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"]
Nov 02 02:09:20.213950 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:20.213916    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ed0600-69b67b547d-cnrvv"]
Nov 02 02:09:20.854570 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:20.854534    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:21.097620 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:21.097578    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:21.226538 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:21.226505    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c785f0a-71f7-432c-900f-bb08d6b82636" path="/var/lib/kubelet/pods/5c785f0a-71f7-432c-900f-bb08d6b82636/volumes"
Nov 02 02:09:21.274029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:21.273996    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:21.428551 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:21.428510    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"]
Nov 02 02:09:21.432247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:21.432217    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed0600-vq8qw"]
Nov 02 02:09:21.865975 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:21.865935    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:23.226491 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:23.226449    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5578ff-6c42-44a3-b4b2-95fac1a297b9" path="/var/lib/kubelet/pods/3a5578ff-6c42-44a3-b4b2-95fac1a297b9/volumes"
Nov 02 02:09:24.036951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:24.036899    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n" podUID="efc0224f-76d8-4279-8739-f6165803bd86" containerName="payload-pre-processing" containerID="cri-o://6fbec0648f9cab7ff3043fdf527884e44daf1e88fa456b6dd6c436c3120d499d" gracePeriod=25
Nov 02 02:09:25.466267 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.466229    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"]
Nov 02 02:09:25.467126 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467101    4912 cpu_manager.go:414] "containerMap: removing container" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main"
Nov 02 02:09:25.467126 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467123    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main"
Nov 02 02:09:25.467310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467150    4912 cpu_manager.go:414] "containerMap: removing container" podUID="7f637d0b-3394-4af9-8237-8519e63188f4" containerName="revoke-keys"
Nov 02 02:09:25.467310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467155    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f637d0b-3394-4af9-8237-8519e63188f4" containerName="revoke-keys"
Nov 02 02:09:25.467310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467179    4912 cpu_manager.go:414] "containerMap: removing container" podUID="0c1c0f26-3660-4874-a458-402952ce2612" containerName="revoke-keys"
Nov 02 02:09:25.467310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467184    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1c0f26-3660-4874-a458-402952ce2612" containerName="revoke-keys"
Nov 02 02:09:25.467310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467205    4912 cpu_manager.go:414] "containerMap: removing container" podUID="5c785f0a-71f7-432c-900f-bb08d6b82636" containerName="maas-api"
Nov 02 02:09:25.467310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467210    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c785f0a-71f7-432c-900f-bb08d6b82636" containerName="maas-api"
Nov 02 02:09:25.467310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467298    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1c0f26-3660-4874-a458-402952ce2612" containerName="revoke-keys"
Nov 02 02:09:25.467310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467308    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c785f0a-71f7-432c-900f-bb08d6b82636" containerName="maas-api"
Nov 02 02:09:25.467310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467315    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24ede2a-7f50-43fc-862b-b27ba373340d" containerName="main"
Nov 02 02:09:25.467595 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.467324    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f637d0b-3394-4af9-8237-8519e63188f4" containerName="revoke-keys"
Nov 02 02:09:25.472177 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.472155    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.520765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.520725    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"]
Nov 02 02:09:25.524782 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.524763    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.566255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.566217    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:09:25.566473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.566437    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:09:25.570719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.570691    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"]
Nov 02 02:09:25.594958 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.594917    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"]
Nov 02 02:09:25.595750 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.595716    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4355d1e2-pjlws\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.595901 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.595793    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4355d1e2-pjlws\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.595973 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.595892    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbfb6b19-trsw4\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.596035 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.596010    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6gn\" (UniqueName: \"kubernetes.io/projected/decdd346-1f3e-445a-b939-e823fe3890af-kube-api-access-nf6gn\") pod \"maas-api-revoke-sub-4355d1e2-pjlws\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.596128 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.596050    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnvtk\" (UniqueName: \"kubernetes.io/projected/eae483f0-4869-4652-8d36-74f0cfb7006d-kube-api-access-cnvtk\") pod \"maas-api-revoke-sub-dbfb6b19-trsw4\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.596128 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.596116    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbfb6b19-trsw4\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.697598 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.697560    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6gn\" (UniqueName: \"kubernetes.io/projected/decdd346-1f3e-445a-b939-e823fe3890af-kube-api-access-nf6gn\") pod \"maas-api-revoke-sub-4355d1e2-pjlws\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.697598 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.697605    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnvtk\" (UniqueName: \"kubernetes.io/projected/eae483f0-4869-4652-8d36-74f0cfb7006d-kube-api-access-cnvtk\") pod \"maas-api-revoke-sub-dbfb6b19-trsw4\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.697821 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.697720    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbfb6b19-trsw4\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.697893 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.697880    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4355d1e2-pjlws\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.697947 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.697934    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4355d1e2-pjlws\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.697984 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.697964    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbfb6b19-trsw4\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.698669 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.698599    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4355d1e2-pjlws\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.698669 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.698606    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbfb6b19-trsw4\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.700432 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.700408    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4355d1e2-pjlws\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.700534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.700408    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbfb6b19-trsw4\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.758361 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.758289    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnvtk\" (UniqueName: \"kubernetes.io/projected/eae483f0-4869-4652-8d36-74f0cfb7006d-kube-api-access-cnvtk\") pod \"maas-api-revoke-sub-dbfb6b19-trsw4\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.758361 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.758287    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6gn\" (UniqueName: \"kubernetes.io/projected/decdd346-1f3e-445a-b939-e823fe3890af-kube-api-access-nf6gn\") pod \"maas-api-revoke-sub-4355d1e2-pjlws\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:25.785993 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.785965    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:25.804884 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.804853    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n" podUID="efc0224f-76d8-4279-8739-f6165803bd86" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:09:25.849164 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:25.849133    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:26.140937 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:26.140906    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:26.167996 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:26.167967    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"]
Nov 02 02:09:26.168190 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:26.168163    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae483f0_4869_4652_8d36_74f0cfb7006d.slice/crio-6aeb04faddcacfda10a83cd1204566f7553a60928dcefd07316d0d051b2db43f WatchSource:0}: Error finding container 6aeb04faddcacfda10a83cd1204566f7553a60928dcefd07316d0d051b2db43f: Status 404 returned error can't find the container with id 6aeb04faddcacfda10a83cd1204566f7553a60928dcefd07316d0d051b2db43f
Nov 02 02:09:26.307807 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:26.307782    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecdd346_1f3e_445a_b939_e823fe3890af.slice/crio-7bece9baa056cd1aae7c0d33e4995057f05224ab6ed1dc833edd07180592dfb3 WatchSource:0}: Error finding container 7bece9baa056cd1aae7c0d33e4995057f05224ab6ed1dc833edd07180592dfb3: Status 404 returned error can't find the container with id 7bece9baa056cd1aae7c0d33e4995057f05224ab6ed1dc833edd07180592dfb3
Nov 02 02:09:26.308618 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:26.308596    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"]
Nov 02 02:09:26.376558 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:26.376524    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:26.553963 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:26.553927    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:26.724170 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:26.724048    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main" containerID="cri-o://5af6cd14af59ac97a39da125cd5e5494252e47e078cca04ac4133002b800b935" gracePeriod=45
Nov 02 02:09:26.762546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:26.762504    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" containerID="cri-o://aef0f75a57a27a157dddad1f30f36fc32166324aa3ded6a39a64f6c598fc9f74" gracePeriod=45
Nov 02 02:09:26.826094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:26.826032    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused"
Nov 02 02:09:26.935810 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:26.935623    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" containerID="cri-o://7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787" gracePeriod=45
Nov 02 02:09:27.063281 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.063227    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4" event={"ID":"eae483f0-4869-4652-8d36-74f0cfb7006d","Type":"ContainerStarted","Data":"207867a32c98cbd44ebf3f8f9661455977be52029d5a4096b50baedc00b0a39c"}
Nov 02 02:09:27.063448 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.063296    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4" event={"ID":"eae483f0-4869-4652-8d36-74f0cfb7006d","Type":"ContainerStarted","Data":"6aeb04faddcacfda10a83cd1204566f7553a60928dcefd07316d0d051b2db43f"}
Nov 02 02:09:27.065137 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.065109    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws" event={"ID":"decdd346-1f3e-445a-b939-e823fe3890af","Type":"ContainerStarted","Data":"2b800bfa25afe4666d65e5bce300094aa6b8f348032e58a6defc2da6409312c2"}
Nov 02 02:09:27.065252 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.065170    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws" event={"ID":"decdd346-1f3e-445a-b939-e823fe3890af","Type":"ContainerStarted","Data":"7bece9baa056cd1aae7c0d33e4995057f05224ab6ed1dc833edd07180592dfb3"}
Nov 02 02:09:27.065657 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.065635    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 02:09:27.067155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.067121    4912 generic.go:356] "Generic (PLEG): container finished" podID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerID="aef0f75a57a27a157dddad1f30f36fc32166324aa3ded6a39a64f6c598fc9f74" exitCode=0
Nov 02 02:09:27.067267 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.067194    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" event={"ID":"2014bd97-f76a-4090-b50e-45448ff71bf9","Type":"ContainerDied","Data":"aef0f75a57a27a157dddad1f30f36fc32166324aa3ded6a39a64f6c598fc9f74"}
Nov 02 02:09:27.069028 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.069007    4912 generic.go:356] "Generic (PLEG): container finished" podID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerID="5af6cd14af59ac97a39da125cd5e5494252e47e078cca04ac4133002b800b935" exitCode=0
Nov 02 02:09:27.069183 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.069039    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c" event={"ID":"36ceb9dd-8c34-4493-b89e-9dfa1a05948d","Type":"ContainerDied","Data":"5af6cd14af59ac97a39da125cd5e5494252e47e078cca04ac4133002b800b935"}
Nov 02 02:09:27.069183 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.069040    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"
Nov 02 02:09:27.069183 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.069082    4912 scope.go:122] "RemoveContainer" containerID="5af6cd14af59ac97a39da125cd5e5494252e47e078cca04ac4133002b800b935"
Nov 02 02:09:27.083916 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.083890    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 02:09:27.119475 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.119445    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-kube-api-access-f842t\" (UniqueName: \"kubernetes.io/projected/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-kube-api-access-f842t\") pod \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") "
Nov 02 02:09:27.119666 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.119530    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tmp-dir\") pod \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") "
Nov 02 02:09:27.119737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.119667    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-model-cache\") pod \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") "
Nov 02 02:09:27.119737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.119705    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-home\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-home\") pod \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") "
Nov 02 02:09:27.119859 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.119779    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-dshm\") pod \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") "
Nov 02 02:09:27.119859 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.119806    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tls-certs\" (UniqueName: \"kubernetes.io/secret/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tls-certs\") pod \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\" (UID: \"36ceb9dd-8c34-4493-b89e-9dfa1a05948d\") "
Nov 02 02:09:27.120226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.120043    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-home" pod "36ceb9dd-8c34-4493-b89e-9dfa1a05948d" (UID: "36ceb9dd-8c34-4493-b89e-9dfa1a05948d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.120226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.120089    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-model-cache" pod "36ceb9dd-8c34-4493-b89e-9dfa1a05948d" (UID: "36ceb9dd-8c34-4493-b89e-9dfa1a05948d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.120408 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.120391    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.120473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.120411    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.120901 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.120870    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tmp-dir" pod "36ceb9dd-8c34-4493-b89e-9dfa1a05948d" (UID: "36ceb9dd-8c34-4493-b89e-9dfa1a05948d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.122906 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.122882    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-dshm" pod "36ceb9dd-8c34-4493-b89e-9dfa1a05948d" (UID: "36ceb9dd-8c34-4493-b89e-9dfa1a05948d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.123268 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.123050    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-kube-api-access-f842t" pod "36ceb9dd-8c34-4493-b89e-9dfa1a05948d" (UID: "36ceb9dd-8c34-4493-b89e-9dfa1a05948d"). InnerVolumeSpecName "kube-api-access-f842t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:27.124091 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.124041    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tls-certs" pod "36ceb9dd-8c34-4493-b89e-9dfa1a05948d" (UID: "36ceb9dd-8c34-4493-b89e-9dfa1a05948d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:27.181601 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.181551    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4" podStartSLOduration=2.181534314 podStartE2EDuration="2.181534314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:27.180533258 +0000 UTC m=+2260.085998616" watchObservedRunningTime="2026-11-02 02:09:27.181534314 +0000 UTC m=+2260.086999670"
Nov 02 02:09:27.220913 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.220885    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-tmp-dir\") pod \"2014bd97-f76a-4090-b50e-45448ff71bf9\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") "
Nov 02 02:09:27.221121 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.220984    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-model-cache\") pod \"2014bd97-f76a-4090-b50e-45448ff71bf9\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") "
Nov 02 02:09:27.221121 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221032    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-dshm\") pod \"2014bd97-f76a-4090-b50e-45448ff71bf9\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") "
Nov 02 02:09:27.221262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221142    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-home\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-home\") pod \"2014bd97-f76a-4090-b50e-45448ff71bf9\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") "
Nov 02 02:09:27.221262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221170    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2014bd97-f76a-4090-b50e-45448ff71bf9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2014bd97-f76a-4090-b50e-45448ff71bf9-tls-certs\") pod \"2014bd97-f76a-4090-b50e-45448ff71bf9\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") "
Nov 02 02:09:27.221262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221203    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-tmp-dir" pod "2014bd97-f76a-4090-b50e-45448ff71bf9" (UID: "2014bd97-f76a-4090-b50e-45448ff71bf9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.221262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221256    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2014bd97-f76a-4090-b50e-45448ff71bf9-kube-api-access-64l9l\" (UniqueName: \"kubernetes.io/projected/2014bd97-f76a-4090-b50e-45448ff71bf9-kube-api-access-64l9l\") pod \"2014bd97-f76a-4090-b50e-45448ff71bf9\" (UID: \"2014bd97-f76a-4090-b50e-45448ff71bf9\") "
Nov 02 02:09:27.221488 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221297    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-model-cache" pod "2014bd97-f76a-4090-b50e-45448ff71bf9" (UID: "2014bd97-f76a-4090-b50e-45448ff71bf9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.221619 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221583    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-home" pod "2014bd97-f76a-4090-b50e-45448ff71bf9" (UID: "2014bd97-f76a-4090-b50e-45448ff71bf9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.221748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221656    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.221748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221678    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.221748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221691    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.221748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221707    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f842t\" (UniqueName: \"kubernetes.io/projected/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-kube-api-access-f842t\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.221748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221722    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ceb9dd-8c34-4493-b89e-9dfa1a05948d-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.221748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.221734    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.249664 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.249632    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-dshm" pod "2014bd97-f76a-4090-b50e-45448ff71bf9" (UID: "2014bd97-f76a-4090-b50e-45448ff71bf9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.249765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.249645    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2014bd97-f76a-4090-b50e-45448ff71bf9-tls-certs" pod "2014bd97-f76a-4090-b50e-45448ff71bf9" (UID: "2014bd97-f76a-4090-b50e-45448ff71bf9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:27.249765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.249748    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2014bd97-f76a-4090-b50e-45448ff71bf9-kube-api-access-64l9l" pod "2014bd97-f76a-4090-b50e-45448ff71bf9" (UID: "2014bd97-f76a-4090-b50e-45448ff71bf9"). InnerVolumeSpecName "kube-api-access-64l9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:27.308519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.308491    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 02:09:27.323075 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.323027    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.323075 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.323075    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2014bd97-f76a-4090-b50e-45448ff71bf9-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.323317 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.323090    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2014bd97-f76a-4090-b50e-45448ff71bf9-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.323317 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.323104    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64l9l\" (UniqueName: \"kubernetes.io/projected/2014bd97-f76a-4090-b50e-45448ff71bf9-kube-api-access-64l9l\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.413591 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.413543    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws" podStartSLOduration=2.413526424 podStartE2EDuration="2.413526424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:27.410145692 +0000 UTC m=+2260.315611050" watchObservedRunningTime="2026-11-02 02:09:27.413526424 +0000 UTC m=+2260.318991845"
Nov 02 02:09:27.424028 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424002    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-dshm\") pod \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") "
Nov 02 02:09:27.424193 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424138    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tmp-dir\") pod \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") "
Nov 02 02:09:27.424279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424193    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tls-certs\" (UniqueName: \"kubernetes.io/secret/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tls-certs\") pod \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") "
Nov 02 02:09:27.424279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424214    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-kube-api-access-jlwb4\" (UniqueName: \"kubernetes.io/projected/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-kube-api-access-jlwb4\") pod \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") "
Nov 02 02:09:27.424279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424259    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-home\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-home\") pod \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") "
Nov 02 02:09:27.424444 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424309    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-model-cache\") pod \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\" (UID: \"96254bb4-5c44-4d15-a6ab-19fd2d79ce62\") "
Nov 02 02:09:27.424498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424467    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tmp-dir" pod "96254bb4-5c44-4d15-a6ab-19fd2d79ce62" (UID: "96254bb4-5c44-4d15-a6ab-19fd2d79ce62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.424617 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424592    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-model-cache" pod "96254bb4-5c44-4d15-a6ab-19fd2d79ce62" (UID: "96254bb4-5c44-4d15-a6ab-19fd2d79ce62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.424788 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424769    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-home" pod "96254bb4-5c44-4d15-a6ab-19fd2d79ce62" (UID: "96254bb4-5c44-4d15-a6ab-19fd2d79ce62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.425036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424863    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.425036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424882    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.425036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.424891    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.426629 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.426600    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tls-certs" pod "96254bb4-5c44-4d15-a6ab-19fd2d79ce62" (UID: "96254bb4-5c44-4d15-a6ab-19fd2d79ce62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:27.426725 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.426625    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-dshm" pod "96254bb4-5c44-4d15-a6ab-19fd2d79ce62" (UID: "96254bb4-5c44-4d15-a6ab-19fd2d79ce62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.426925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.426907    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-kube-api-access-jlwb4" pod "96254bb4-5c44-4d15-a6ab-19fd2d79ce62" (UID: "96254bb4-5c44-4d15-a6ab-19fd2d79ce62"). InnerVolumeSpecName "kube-api-access-jlwb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:27.525690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.525606    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.525690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.525635    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.525690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.525645    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwb4\" (UniqueName: \"kubernetes.io/projected/96254bb4-5c44-4d15-a6ab-19fd2d79ce62-kube-api-access-jlwb4\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.758227 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.758189    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"]
Nov 02 02:09:27.762264 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.762231    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-w3-061fab-kserve-df579b748-28b2c"]
Nov 02 02:09:27.819542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.819443    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj" podUID="956e87a3-da69-4672-82ec-772ed56c413a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:09:27.819542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:27.819518    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx" podUID="633f0926-b18e-4ed5-b916-268160edee9a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:09:28.082922 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.082828    4912 generic.go:356] "Generic (PLEG): container finished" podID="eae483f0-4869-4652-8d36-74f0cfb7006d" containerID="207867a32c98cbd44ebf3f8f9661455977be52029d5a4096b50baedc00b0a39c" exitCode=0
Nov 02 02:09:28.083115 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.082923    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4" event={"ID":"eae483f0-4869-4652-8d36-74f0cfb7006d","Type":"ContainerDied","Data":"207867a32c98cbd44ebf3f8f9661455977be52029d5a4096b50baedc00b0a39c"}
Nov 02 02:09:28.084835 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.084811    4912 generic.go:356] "Generic (PLEG): container finished" podID="decdd346-1f3e-445a-b939-e823fe3890af" containerID="2b800bfa25afe4666d65e5bce300094aa6b8f348032e58a6defc2da6409312c2" exitCode=0
Nov 02 02:09:28.084964 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.084901    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws" event={"ID":"decdd346-1f3e-445a-b939-e823fe3890af","Type":"ContainerDied","Data":"2b800bfa25afe4666d65e5bce300094aa6b8f348032e58a6defc2da6409312c2"}
Nov 02 02:09:28.086510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.086495    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"
Nov 02 02:09:28.086604 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.086507    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs" event={"ID":"2014bd97-f76a-4090-b50e-45448ff71bf9","Type":"ContainerDied","Data":"7af7398d7b87f8fd32b8021e85c62cecc5a2c7ed9100b435fd69716ea2921ede"}
Nov 02 02:09:28.086604 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.086551    4912 scope.go:122] "RemoveContainer" containerID="aef0f75a57a27a157dddad1f30f36fc32166324aa3ded6a39a64f6c598fc9f74"
Nov 02 02:09:28.088665 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.088642    4912 generic.go:356] "Generic (PLEG): container finished" podID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerID="7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787" exitCode=0
Nov 02 02:09:28.088780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.088678    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" event={"ID":"96254bb4-5c44-4d15-a6ab-19fd2d79ce62","Type":"ContainerDied","Data":"7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787"}
Nov 02 02:09:28.088780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.088702    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" event={"ID":"96254bb4-5c44-4d15-a6ab-19fd2d79ce62","Type":"ContainerDied","Data":"ec52d4db2a57d81fdc489ccf87c0b01527af7efdce6af1626da7b615c93fa3cd"}
Nov 02 02:09:28.088780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.088755    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"
Nov 02 02:09:28.101715 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.101700    4912 scope.go:122] "RemoveContainer" containerID="7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787"
Nov 02 02:09:28.113004 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.112981    4912 scope.go:122] "RemoveContainer" containerID="7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787"
Nov 02 02:09:28.113322 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:28.113297    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787\": container with ID starting with 7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787 not found: ID does not exist" containerID="7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787"
Nov 02 02:09:28.113386 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.113326    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787"} err="failed to get container status \"7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787\": rpc error: code = NotFound desc = could not find container \"7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787\": container with ID starting with 7246d1e11c0424182483d29a072f6a4b3f61e0d0b1ed19f1384f055a67310787 not found: ID does not exist"
Nov 02 02:09:28.125956 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.125933    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:28.168992 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.168947    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Nov 02 02:09:28.310256 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.310222    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"]
Nov 02 02:09:28.314083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.314033    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/distinct-2-w3-061fab-kserve-98567c679-w5c5l"]
Nov 02 02:09:28.355149 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.355023    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:28.388326 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.388292    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"]
Nov 02 02:09:28.388574 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.388524    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy" containerID="cri-o://24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058" gracePeriod=30
Nov 02 02:09:28.422963 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.422931    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"]
Nov 02 02:09:28.428099 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.428071    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/simulator-w3-061fab-kserve-797df4ff5b-wbqrs"]
Nov 02 02:09:28.576079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:28.570539    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:29.235472 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.235429    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" path="/var/lib/kubelet/pods/2014bd97-f76a-4090-b50e-45448ff71bf9/volumes"
Nov 02 02:09:29.236136 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.236110    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" path="/var/lib/kubelet/pods/36ceb9dd-8c34-4493-b89e-9dfa1a05948d/volumes"
Nov 02 02:09:29.236582 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.236563    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" path="/var/lib/kubelet/pods/96254bb4-5c44-4d15-a6ab-19fd2d79ce62/volumes"
Nov 02 02:09:29.296515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.296470    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:29.332381 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.332352    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:29.372897 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.372864    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-xks-ca\") pod \"eae483f0-4869-4652-8d36-74f0cfb7006d\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") "
Nov 02 02:09:29.373210 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.372962    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae483f0-4869-4652-8d36-74f0cfb7006d-kube-api-access-cnvtk\" (UniqueName: \"kubernetes.io/projected/eae483f0-4869-4652-8d36-74f0cfb7006d-kube-api-access-cnvtk\") pod \"eae483f0-4869-4652-8d36-74f0cfb7006d\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") "
Nov 02 02:09:29.373210 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.373147    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-openshift-ca\") pod \"eae483f0-4869-4652-8d36-74f0cfb7006d\" (UID: \"eae483f0-4869-4652-8d36-74f0cfb7006d\") "
Nov 02 02:09:29.374172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.373658    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-openshift-ca" pod "eae483f0-4869-4652-8d36-74f0cfb7006d" (UID: "eae483f0-4869-4652-8d36-74f0cfb7006d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:29.375448 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.375426    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-xks-ca" pod "eae483f0-4869-4652-8d36-74f0cfb7006d" (UID: "eae483f0-4869-4652-8d36-74f0cfb7006d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:29.375692 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.375669    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae483f0-4869-4652-8d36-74f0cfb7006d-kube-api-access-cnvtk" pod "eae483f0-4869-4652-8d36-74f0cfb7006d" (UID: "eae483f0-4869-4652-8d36-74f0cfb7006d"). InnerVolumeSpecName "kube-api-access-cnvtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:29.475002 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.474972    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decdd346-1f3e-445a-b939-e823fe3890af-kube-api-access-nf6gn\" (UniqueName: \"kubernetes.io/projected/decdd346-1f3e-445a-b939-e823fe3890af-kube-api-access-nf6gn\") pod \"decdd346-1f3e-445a-b939-e823fe3890af\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") "
Nov 02 02:09:29.475206 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.475023    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-xks-ca\") pod \"decdd346-1f3e-445a-b939-e823fe3890af\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") "
Nov 02 02:09:29.475206 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.475043    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-openshift-ca\") pod \"decdd346-1f3e-445a-b939-e823fe3890af\" (UID: \"decdd346-1f3e-445a-b939-e823fe3890af\") "
Nov 02 02:09:29.475469 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.475448    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:29.475534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.475470    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-openshift-ca" pod "decdd346-1f3e-445a-b939-e823fe3890af" (UID: "decdd346-1f3e-445a-b939-e823fe3890af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:29.475534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.475478    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae483f0-4869-4652-8d36-74f0cfb7006d-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:29.475644 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.475548    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnvtk\" (UniqueName: \"kubernetes.io/projected/eae483f0-4869-4652-8d36-74f0cfb7006d-kube-api-access-cnvtk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:29.477532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.477503    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-xks-ca" pod "decdd346-1f3e-445a-b939-e823fe3890af" (UID: "decdd346-1f3e-445a-b939-e823fe3890af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:29.477532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.477509    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decdd346-1f3e-445a-b939-e823fe3890af-kube-api-access-nf6gn" pod "decdd346-1f3e-445a-b939-e823fe3890af" (UID: "decdd346-1f3e-445a-b939-e823fe3890af"). InnerVolumeSpecName "kube-api-access-nf6gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:29.577007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.576972    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf6gn\" (UniqueName: \"kubernetes.io/projected/decdd346-1f3e-445a-b939-e823fe3890af-kube-api-access-nf6gn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:29.577007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.577002    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:29.577007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:29.577014    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decdd346-1f3e-445a-b939-e823fe3890af-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:30.115535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.115490    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4" event={"ID":"eae483f0-4869-4652-8d36-74f0cfb7006d","Type":"ContainerDied","Data":"6aeb04faddcacfda10a83cd1204566f7553a60928dcefd07316d0d051b2db43f"}
Nov 02 02:09:30.115535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.115528    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"
Nov 02 02:09:30.115824 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.115560    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aeb04faddcacfda10a83cd1204566f7553a60928dcefd07316d0d051b2db43f"
Nov 02 02:09:30.117140 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.117114    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"
Nov 02 02:09:30.117259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.117153    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws" event={"ID":"decdd346-1f3e-445a-b939-e823fe3890af","Type":"ContainerDied","Data":"7bece9baa056cd1aae7c0d33e4995057f05224ab6ed1dc833edd07180592dfb3"}
Nov 02 02:09:30.117259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.117204    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bece9baa056cd1aae7c0d33e4995057f05224ab6ed1dc833edd07180592dfb3"
Nov 02 02:09:30.781372 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.781299    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 02:09:30.892299 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.892269    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956e87a3-da69-4672-82ec-772ed56c413a-kube-api-access-d8xjz\" (UniqueName: \"kubernetes.io/projected/956e87a3-da69-4672-82ec-772ed56c413a-kube-api-access-d8xjz\") pod \"956e87a3-da69-4672-82ec-772ed56c413a\" (UID: \"956e87a3-da69-4672-82ec-772ed56c413a\") "
Nov 02 02:09:30.892501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.892486    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956e87a3-da69-4672-82ec-772ed56c413a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956e87a3-da69-4672-82ec-772ed56c413a-plugins-config-volume\") pod \"956e87a3-da69-4672-82ec-772ed56c413a\" (UID: \"956e87a3-da69-4672-82ec-772ed56c413a\") "
Nov 02 02:09:30.892878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.892850    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956e87a3-da69-4672-82ec-772ed56c413a-plugins-config-volume" pod "956e87a3-da69-4672-82ec-772ed56c413a" (UID: "956e87a3-da69-4672-82ec-772ed56c413a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:30.893052 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.893032    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956e87a3-da69-4672-82ec-772ed56c413a-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:30.894744 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.894718    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956e87a3-da69-4672-82ec-772ed56c413a-kube-api-access-d8xjz" pod "956e87a3-da69-4672-82ec-772ed56c413a" (UID: "956e87a3-da69-4672-82ec-772ed56c413a"). InnerVolumeSpecName "kube-api-access-d8xjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:30.994211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:30.994179    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8xjz\" (UniqueName: \"kubernetes.io/projected/956e87a3-da69-4672-82ec-772ed56c413a-kube-api-access-d8xjz\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:31.091440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.091411    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 02:09:31.127118 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.127084    4912 generic.go:356] "Generic (PLEG): container finished" podID="633f0926-b18e-4ed5-b916-268160edee9a" containerID="bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85" exitCode=0
Nov 02 02:09:31.127269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.127127    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx" event={"ID":"633f0926-b18e-4ed5-b916-268160edee9a","Type":"ContainerDied","Data":"bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85"}
Nov 02 02:09:31.127269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.127160    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"
Nov 02 02:09:31.127269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.127201    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx" event={"ID":"633f0926-b18e-4ed5-b916-268160edee9a","Type":"ContainerDied","Data":"68e5a38df6282e4158e03d747119c53a902fc94426bc166bd40dbc62a0edc543"}
Nov 02 02:09:31.127269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.127231    4912 scope.go:122] "RemoveContainer" containerID="bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85"
Nov 02 02:09:31.129156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.129130    4912 generic.go:356] "Generic (PLEG): container finished" podID="956e87a3-da69-4672-82ec-772ed56c413a" containerID="03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290" exitCode=0
Nov 02 02:09:31.129278 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.129169    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"
Nov 02 02:09:31.129278 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.129169    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj" event={"ID":"956e87a3-da69-4672-82ec-772ed56c413a","Type":"ContainerDied","Data":"03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290"}
Nov 02 02:09:31.129278 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.129212    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj" event={"ID":"956e87a3-da69-4672-82ec-772ed56c413a","Type":"ContainerDied","Data":"d3c81cfc112cc27faa42e8a93f367c3639283b34a3a237ce7ac9017f6ea2fc6b"}
Nov 02 02:09:31.140584 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.140567    4912 scope.go:122] "RemoveContainer" containerID="bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85"
Nov 02 02:09:31.140851 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:31.140834    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85\": container with ID starting with bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85 not found: ID does not exist" containerID="bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85"
Nov 02 02:09:31.140929 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.140860    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85"} err="failed to get container status \"bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85\": rpc error: code = NotFound desc = could not find container \"bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85\": container with ID starting with bb44065ee6274b50e1cb7397857804df22e5bddb03a5520feda5f566fa556c85 not found: ID does not exist"
Nov 02 02:09:31.140929 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.140885    4912 scope.go:122] "RemoveContainer" containerID="03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290"
Nov 02 02:09:31.151155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.151128    4912 scope.go:122] "RemoveContainer" containerID="03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290"
Nov 02 02:09:31.151451 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:31.151427    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290\": container with ID starting with 03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290 not found: ID does not exist" containerID="03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290"
Nov 02 02:09:31.151551 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.151461    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290"} err="failed to get container status \"03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290\": rpc error: code = NotFound desc = could not find container \"03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290\": container with ID starting with 03c0d8c3fc6342b26199b025954b65c2493327e53d521c65c06ed88458485290 not found: ID does not exist"
Nov 02 02:09:31.196673 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.196642    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633f0926-b18e-4ed5-b916-268160edee9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633f0926-b18e-4ed5-b916-268160edee9a-plugins-config-volume\") pod \"633f0926-b18e-4ed5-b916-268160edee9a\" (UID: \"633f0926-b18e-4ed5-b916-268160edee9a\") "
Nov 02 02:09:31.196859 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.196694    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633f0926-b18e-4ed5-b916-268160edee9a-kube-api-access-74frn\" (UniqueName: \"kubernetes.io/projected/633f0926-b18e-4ed5-b916-268160edee9a-kube-api-access-74frn\") pod \"633f0926-b18e-4ed5-b916-268160edee9a\" (UID: \"633f0926-b18e-4ed5-b916-268160edee9a\") "
Nov 02 02:09:31.197042 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.197016    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633f0926-b18e-4ed5-b916-268160edee9a-plugins-config-volume" pod "633f0926-b18e-4ed5-b916-268160edee9a" (UID: "633f0926-b18e-4ed5-b916-268160edee9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:31.197301 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.197280    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633f0926-b18e-4ed5-b916-268160edee9a-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:31.199045 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.199016    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633f0926-b18e-4ed5-b916-268160edee9a-kube-api-access-74frn" pod "633f0926-b18e-4ed5-b916-268160edee9a" (UID: "633f0926-b18e-4ed5-b916-268160edee9a"). InnerVolumeSpecName "kube-api-access-74frn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:31.298913 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.298878    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74frn\" (UniqueName: \"kubernetes.io/projected/633f0926-b18e-4ed5-b916-268160edee9a-kube-api-access-74frn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:31.382260 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.382217    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"]
Nov 02 02:09:31.388916 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.388879    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-55bc84-76f6bf9fcf-gzznj"]
Nov 02 02:09:31.455744 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.455699    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:31.556987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.556940    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"]
Nov 02 02:09:31.563175 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:31.563140    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-55bc84-7b7b8854c5-rsrmx"]
Nov 02 02:09:33.225543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:33.225509    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633f0926-b18e-4ed5-b916-268160edee9a" path="/var/lib/kubelet/pods/633f0926-b18e-4ed5-b916-268160edee9a/volumes"
Nov 02 02:09:33.225968 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:33.225890    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956e87a3-da69-4672-82ec-772ed56c413a" path="/var/lib/kubelet/pods/956e87a3-da69-4672-82ec-772ed56c413a/volumes"
Nov 02 02:09:34.135650 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.135628    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 02:09:34.160145 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.160049    4912 generic.go:356] "Generic (PLEG): container finished" podID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerID="24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058" exitCode=0
Nov 02 02:09:34.160369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.160351    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" event={"ID":"10b205f3-b937-4bd5-80f8-3c2982a5fd3e","Type":"ContainerDied","Data":"24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058"}
Nov 02 02:09:34.160428 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.160407    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b" event={"ID":"10b205f3-b937-4bd5-80f8-3c2982a5fd3e","Type":"ContainerDied","Data":"7516cab8d3155d684bb8f27896d6b3ffd88361c2f5c892c6b9fa1de0df6154a5"}
Nov 02 02:09:34.160485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.160436    4912 scope.go:122] "RemoveContainer" containerID="24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058"
Nov 02 02:09:34.160595 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.160583    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"
Nov 02 02:09:34.174395 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.174369    4912 scope.go:122] "RemoveContainer" containerID="c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa"
Nov 02 02:09:34.186946 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.186923    4912 scope.go:122] "RemoveContainer" containerID="24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058"
Nov 02 02:09:34.187308 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:34.187274    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058\": container with ID starting with 24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058 not found: ID does not exist" containerID="24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058"
Nov 02 02:09:34.187393 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.187314    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058"} err="failed to get container status \"24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058\": rpc error: code = NotFound desc = could not find container \"24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058\": container with ID starting with 24cb4ba6507d5439b770b1474b2fb37133108da1b00eeb347017de4fc2ccb058 not found: ID does not exist"
Nov 02 02:09:34.187393 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.187339    4912 scope.go:122] "RemoveContainer" containerID="c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa"
Nov 02 02:09:34.187660 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:34.187639    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa\": container with ID starting with c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa not found: ID does not exist" containerID="c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa"
Nov 02 02:09:34.187740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.187669    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa"} err="failed to get container status \"c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa\": rpc error: code = NotFound desc = could not find container \"c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa\": container with ID starting with c62031751ae3a7d748cff82d7ab9958da7765b34368bf7b4589f4cd2091157aa not found: ID does not exist"
Nov 02 02:09:34.233371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233346    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-credential-socket\") pod \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") "
Nov 02 02:09:34.233799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233383    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istiod-ca-cert\") pod \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") "
Nov 02 02:09:34.233799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233434    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-podinfo\") pod \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") "
Nov 02 02:09:34.233799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233501    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-kube-api-access-rvvwq\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-kube-api-access-rvvwq\") pod \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") "
Nov 02 02:09:34.233799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233529    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-certs\") pod \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") "
Nov 02 02:09:34.233799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233567    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-envoy\") pod \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") "
Nov 02 02:09:34.233799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233622    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-socket\") pod \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") "
Nov 02 02:09:34.233799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233626    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-credential-socket" pod "10b205f3-b937-4bd5-80f8-3c2982a5fd3e" (UID: "10b205f3-b937-4bd5-80f8-3c2982a5fd3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:34.233799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233646    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-data\") pod \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") "
Nov 02 02:09:34.233799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233709    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-token\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-token\") pod \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\" (UID: \"10b205f3-b937-4bd5-80f8-3c2982a5fd3e\") "
Nov 02 02:09:34.234531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233816    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istiod-ca-cert" pod "10b205f3-b937-4bd5-80f8-3c2982a5fd3e" (UID: "10b205f3-b937-4bd5-80f8-3c2982a5fd3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:34.234531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.233912    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-socket" pod "10b205f3-b937-4bd5-80f8-3c2982a5fd3e" (UID: "10b205f3-b937-4bd5-80f8-3c2982a5fd3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:34.234531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.234020    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-data" pod "10b205f3-b937-4bd5-80f8-3c2982a5fd3e" (UID: "10b205f3-b937-4bd5-80f8-3c2982a5fd3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:34.234531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.234039    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-certs" pod "10b205f3-b937-4bd5-80f8-3c2982a5fd3e" (UID: "10b205f3-b937-4bd5-80f8-3c2982a5fd3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:34.234531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.234217    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.234531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.234237    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.234531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.234254    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.234531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.234272    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.234531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.234287    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.235905 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.235882    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-envoy" pod "10b205f3-b937-4bd5-80f8-3c2982a5fd3e" (UID: "10b205f3-b937-4bd5-80f8-3c2982a5fd3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:34.236016 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.235934    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-kube-api-access-rvvwq" pod "10b205f3-b937-4bd5-80f8-3c2982a5fd3e" (UID: "10b205f3-b937-4bd5-80f8-3c2982a5fd3e"). InnerVolumeSpecName "kube-api-access-rvvwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:34.236105 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.236023    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-podinfo" pod "10b205f3-b937-4bd5-80f8-3c2982a5fd3e" (UID: "10b205f3-b937-4bd5-80f8-3c2982a5fd3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:09:34.236150 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.236131    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-token" pod "10b205f3-b937-4bd5-80f8-3c2982a5fd3e" (UID: "10b205f3-b937-4bd5-80f8-3c2982a5fd3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:34.335713 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.335670    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.335713 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.335709    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvwq\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-kube-api-access-rvvwq\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.335974 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.335725    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.335974 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.335740    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b205f3-b937-4bd5-80f8-3c2982a5fd3e-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.591684 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.591645    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"]
Nov 02 02:09:34.595248 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:34.595219    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-m644b"]
Nov 02 02:09:35.226340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:35.226301    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" path="/var/lib/kubelet/pods/10b205f3-b937-4bd5-80f8-3c2982a5fd3e/volumes"
Nov 02 02:09:35.487447 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:35.487353    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:35.680081 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:35.669821    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:35.805156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:35.805044    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n" podUID="efc0224f-76d8-4279-8739-f6165803bd86" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:09:36.282656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:36.282525    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:40.490992 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.490905    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7028-ba556b"]
Nov 02 02:09:40.491637 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491617    4912 cpu_manager.go:414] "containerMap: removing container" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy"
Nov 02 02:09:40.491694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491639    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy"
Nov 02 02:09:40.491694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491654    4912 cpu_manager.go:414] "containerMap: removing container" podUID="633f0926-b18e-4ed5-b916-268160edee9a" containerName="payload-pre-processing"
Nov 02 02:09:40.491694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491661    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633f0926-b18e-4ed5-b916-268160edee9a" containerName="payload-pre-processing"
Nov 02 02:09:40.491694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491689    4912 cpu_manager.go:414] "containerMap: removing container" podUID="eae483f0-4869-4652-8d36-74f0cfb7006d" containerName="revoke-keys"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491695    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae483f0-4869-4652-8d36-74f0cfb7006d" containerName="revoke-keys"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491706    4912 cpu_manager.go:414] "containerMap: removing container" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491712    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491722    4912 cpu_manager.go:414] "containerMap: removing container" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491728    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491763    4912 cpu_manager.go:414] "containerMap: removing container" podUID="956e87a3-da69-4672-82ec-772ed56c413a" containerName="payload-processing"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491770    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956e87a3-da69-4672-82ec-772ed56c413a" containerName="payload-processing"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491780    4912 cpu_manager.go:414] "containerMap: removing container" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491786    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491802    4912 cpu_manager.go:414] "containerMap: removing container" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main"
Nov 02 02:09:40.491814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491808    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491822    4912 cpu_manager.go:414] "containerMap: removing container" podUID="decdd346-1f3e-445a-b939-e823fe3890af" containerName="revoke-keys"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491829    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decdd346-1f3e-445a-b939-e823fe3890af" containerName="revoke-keys"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491953    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ceb9dd-8c34-4493-b89e-9dfa1a05948d" containerName="main"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491967    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="decdd346-1f3e-445a-b939-e823fe3890af" containerName="revoke-keys"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491977    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491984    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="633f0926-b18e-4ed5-b916-268160edee9a" containerName="payload-pre-processing"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.491995    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae483f0-4869-4652-8d36-74f0cfb7006d" containerName="revoke-keys"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.492003    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="96254bb4-5c44-4d15-a6ab-19fd2d79ce62" containerName="main"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.492016    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="956e87a3-da69-4672-82ec-772ed56c413a" containerName="payload-processing"
Nov 02 02:09:40.492172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.492022    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="2014bd97-f76a-4090-b50e-45448ff71bf9" containerName="main"
Nov 02 02:09:40.496005 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.495969    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:40.538550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.538510    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrjz\" (UniqueName: \"kubernetes.io/projected/be46f62a-1a48-4701-bac1-523b4398f80c-kube-api-access-bzrjz\") pod \"test-curl-7028-ba556b\" (UID: \"be46f62a-1a48-4701-bac1-523b4398f80c\") " pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:40.548626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.548597    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-ba556b"]
Nov 02 02:09:40.593651 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.593623    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v4n9t\""
Nov 02 02:09:40.639756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.639718    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrjz\" (UniqueName: \"kubernetes.io/projected/be46f62a-1a48-4701-bac1-523b4398f80c-kube-api-access-bzrjz\") pod \"test-curl-7028-ba556b\" (UID: \"be46f62a-1a48-4701-bac1-523b4398f80c\") " pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:40.700800 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.700757    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrjz\" (UniqueName: \"kubernetes.io/projected/be46f62a-1a48-4701-bac1-523b4398f80c-kube-api-access-bzrjz\") pod \"test-curl-7028-ba556b\" (UID: \"be46f62a-1a48-4701-bac1-523b4398f80c\") " pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:40.811192 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:40.811104    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:41.191173 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:41.191144    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-ba556b"]
Nov 02 02:09:41.191604 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:41.191579    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe46f62a_1a48_4701_bac1_523b4398f80c.slice/crio-9f04609f44ca5b4ee2cc9d8a4bee86cded13498c1d330d29669df8f4f9f40bbd WatchSource:0}: Error finding container 9f04609f44ca5b4ee2cc9d8a4bee86cded13498c1d330d29669df8f4f9f40bbd: Status 404 returned error can't find the container with id 9f04609f44ca5b4ee2cc9d8a4bee86cded13498c1d330d29669df8f4f9f40bbd
Nov 02 02:09:41.247882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:41.247853    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-ba556b" event={"ID":"be46f62a-1a48-4701-bac1-523b4398f80c","Type":"ContainerStarted","Data":"9f04609f44ca5b4ee2cc9d8a4bee86cded13498c1d330d29669df8f4f9f40bbd"}
Nov 02 02:09:41.505667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:41.505580    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:41.706583 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:41.706545    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:42.138971 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:42.138933    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:09:42.259532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:42.259474    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-ba556b" event={"ID":"be46f62a-1a48-4701-bac1-523b4398f80c","Type":"ContainerStarted","Data":"60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d"}
Nov 02 02:09:42.372785 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:42.372726    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7028-ba556b" podStartSLOduration=2.372709248 podStartE2EDuration="2.372709248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:42.372212362 +0000 UTC m=+2275.277677739" watchObservedRunningTime="2026-11-02 02:09:42.372709248 +0000 UTC m=+2275.278174608"
Nov 02 02:09:43.894683 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:43.894645    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7028-ba556b"]
Nov 02 02:09:43.920781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:43.920744    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7028-ba556b"]
Nov 02 02:09:44.285572 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.285543    4912 generic.go:356] "Generic (PLEG): container finished" podID="efc0224f-76d8-4279-8739-f6165803bd86" containerID="6fbec0648f9cab7ff3043fdf527884e44daf1e88fa456b6dd6c436c3120d499d" exitCode=0
Nov 02 02:09:44.285887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.285864    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7028-ba556b" podUID="be46f62a-1a48-4701-bac1-523b4398f80c" containerName="test-curl-7028-ba556b" containerID="cri-o://60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d" gracePeriod=2
Nov 02 02:09:44.286001 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.285980    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n" event={"ID":"efc0224f-76d8-4279-8739-f6165803bd86","Type":"ContainerDied","Data":"6fbec0648f9cab7ff3043fdf527884e44daf1e88fa456b6dd6c436c3120d499d"}
Nov 02 02:09:44.316326 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.316301    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 02:09:44.337086 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:44.337026    4912 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7028-ba556b\" is forbidden: User \"system:node:ip-10-0-135-33.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-33.us-west-1.compute.internal' and this object" podUID="be46f62a-1a48-4701-bac1-523b4398f80c" pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:44.383304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.383261    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc0224f-76d8-4279-8739-f6165803bd86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc0224f-76d8-4279-8739-f6165803bd86-plugins-config-volume\") pod \"efc0224f-76d8-4279-8739-f6165803bd86\" (UID: \"efc0224f-76d8-4279-8739-f6165803bd86\") "
Nov 02 02:09:44.383563 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.383543    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc0224f-76d8-4279-8739-f6165803bd86-kube-api-access-z6fj7\" (UniqueName: \"kubernetes.io/projected/efc0224f-76d8-4279-8739-f6165803bd86-kube-api-access-z6fj7\") pod \"efc0224f-76d8-4279-8739-f6165803bd86\" (UID: \"efc0224f-76d8-4279-8739-f6165803bd86\") "
Nov 02 02:09:44.383666 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.383643    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc0224f-76d8-4279-8739-f6165803bd86-plugins-config-volume" pod "efc0224f-76d8-4279-8739-f6165803bd86" (UID: "efc0224f-76d8-4279-8739-f6165803bd86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:44.383927 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.383912    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc0224f-76d8-4279-8739-f6165803bd86-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:44.385780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.385752    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc0224f-76d8-4279-8739-f6165803bd86-kube-api-access-z6fj7" pod "efc0224f-76d8-4279-8739-f6165803bd86" (UID: "efc0224f-76d8-4279-8739-f6165803bd86"). InnerVolumeSpecName "kube-api-access-z6fj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:44.461550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.461468    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"]
Nov 02 02:09:44.462048 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.462032    4912 cpu_manager.go:414] "containerMap: removing container" podUID="efc0224f-76d8-4279-8739-f6165803bd86" containerName="payload-pre-processing"
Nov 02 02:09:44.462048 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.462048    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc0224f-76d8-4279-8739-f6165803bd86" containerName="payload-pre-processing"
Nov 02 02:09:44.462216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.462095    4912 cpu_manager.go:414] "containerMap: removing container" podUID="be46f62a-1a48-4701-bac1-523b4398f80c" containerName="test-curl-7028-ba556b"
Nov 02 02:09:44.462216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.462101    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be46f62a-1a48-4701-bac1-523b4398f80c" containerName="test-curl-7028-ba556b"
Nov 02 02:09:44.462216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.462203    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="be46f62a-1a48-4701-bac1-523b4398f80c" containerName="test-curl-7028-ba556b"
Nov 02 02:09:44.462216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.462215    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b205f3-b937-4bd5-80f8-3c2982a5fd3e" containerName="istio-proxy"
Nov 02 02:09:44.462439 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.462226    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc0224f-76d8-4279-8739-f6165803bd86" containerName="payload-pre-processing"
Nov 02 02:09:44.465832 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.465810    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.484953 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.484907    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n959x\" (UniqueName: \"kubernetes.io/projected/8729baf1-9239-482b-807e-1d1c23ac7b51-kube-api-access-n959x\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.485139 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.485113    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.485203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.485167    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.485285 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.485268    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6fj7\" (UniqueName: \"kubernetes.io/projected/efc0224f-76d8-4279-8739-f6165803bd86-kube-api-access-z6fj7\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:44.505509 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:44.505472    4912 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7028-ba556b\" is forbidden: User \"system:node:ip-10-0-135-33.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-33.us-west-1.compute.internal' and this object" podUID="be46f62a-1a48-4701-bac1-523b4398f80c" pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:44.518808 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.518772    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:09:44.518978 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.518827    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:09:44.525103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.521381    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"]
Nov 02 02:09:44.556253 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:44.556214    4912 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7028-ba556b\" is forbidden: User \"system:node:ip-10-0-135-33.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-33.us-west-1.compute.internal' and this object" podUID="be46f62a-1a48-4701-bac1-523b4398f80c" pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:44.586022 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.585989    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n959x\" (UniqueName: \"kubernetes.io/projected/8729baf1-9239-482b-807e-1d1c23ac7b51-kube-api-access-n959x\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.586280 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.586227    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.586280 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.586262    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.586932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.586906    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.588834 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.588811    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.646910 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.646880    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n959x\" (UniqueName: \"kubernetes.io/projected/8729baf1-9239-482b-807e-1d1c23ac7b51-kube-api-access-n959x\") pod \"maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:44.783975 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:44.783870    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:45.168976 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:45.168944    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8729baf1_9239_482b_807e_1d1c23ac7b51.slice/crio-8912eb42c64ebb224ed02f6b0b60541210cf92252df9654e466a5865f2cdab0e WatchSource:0}: Error finding container 8912eb42c64ebb224ed02f6b0b60541210cf92252df9654e466a5865f2cdab0e: Status 404 returned error can't find the container with id 8912eb42c64ebb224ed02f6b0b60541210cf92252df9654e466a5865f2cdab0e
Nov 02 02:09:45.170004 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:45.169977    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"]
Nov 02 02:09:45.295692 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:45.295664    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"
Nov 02 02:09:45.295692 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:45.295676    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n" event={"ID":"efc0224f-76d8-4279-8739-f6165803bd86","Type":"ContainerDied","Data":"dfd1501d5fe574ce2c4aa338f42c674686e5578acfa59935f3c731484d0397cb"}
Nov 02 02:09:45.295957 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:45.295743    4912 scope.go:122] "RemoveContainer" containerID="6fbec0648f9cab7ff3043fdf527884e44daf1e88fa456b6dd6c436c3120d499d"
Nov 02 02:09:45.297870 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:45.297710    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh" event={"ID":"8729baf1-9239-482b-807e-1d1c23ac7b51","Type":"ContainerStarted","Data":"8912eb42c64ebb224ed02f6b0b60541210cf92252df9654e466a5865f2cdab0e"}
Nov 02 02:09:45.474648 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:45.474608    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"]
Nov 02 02:09:45.516191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:45.516161    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-061fab-6fdfc66dff-2778n"]
Nov 02 02:09:46.210522 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.210487    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"]
Nov 02 02:09:46.214620 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.214597    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.305903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.305863    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-metrics-certs\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.306069 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.305925    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.306069 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.305942    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kljg6\" (UniqueName: \"kubernetes.io/projected/d1ce6436-0b1f-4f8d-8e52-112531e29105-kube-api-access-kljg6\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.306652 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.306611    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce6436-0b1f-4f8d-8e52-112531e29105-openshift-service-ca\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.310154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.310130    4912 generic.go:356] "Generic (PLEG): container finished" podID="8729baf1-9239-482b-807e-1d1c23ac7b51" containerID="4a66f7bda6d21091bdc10d49a7f904e2e4f931861a45585caf21db740f940be0" exitCode=0
Nov 02 02:09:46.310273 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.310214    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh" event={"ID":"8729baf1-9239-482b-807e-1d1c23ac7b51","Type":"ContainerDied","Data":"4a66f7bda6d21091bdc10d49a7f904e2e4f931861a45585caf21db740f940be0"}
Nov 02 02:09:46.319290 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.319271    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-092079-serving-cert\""
Nov 02 02:09:46.319290 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.319283    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2n4c5\""
Nov 02 02:09:46.319483 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.319451    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 02:09:46.323629 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.323584    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"]
Nov 02 02:09:46.407867 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.407832    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce6436-0b1f-4f8d-8e52-112531e29105-openshift-service-ca\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.408127 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.407899    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-metrics-certs\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.408127 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.408005    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.408127 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.408042    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kljg6\" (UniqueName: \"kubernetes.io/projected/d1ce6436-0b1f-4f8d-8e52-112531e29105-kube-api-access-kljg6\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.408297 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:46.408174    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-092079-serving-cert: secret "maas-api-e2e-api-a-092079-serving-cert" not found
Nov 02 02:09:46.408297 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:46.408264    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls podName:d1ce6436-0b1f-4f8d-8e52-112531e29105 nodeName:}" failed. No retries permitted until 2026-11-02 02:09:46.908242981 +0000 UTC m=+2279.813708333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls") pod "maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh" (UID: "d1ce6436-0b1f-4f8d-8e52-112531e29105") : secret "maas-api-e2e-api-a-092079-serving-cert" not found
Nov 02 02:09:46.408634 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.408614    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce6436-0b1f-4f8d-8e52-112531e29105-openshift-service-ca\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.410526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.410508    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-metrics-certs\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.431668 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.431634    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:46.504133 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.503090    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kljg6\" (UniqueName: \"kubernetes.io/projected/d1ce6436-0b1f-4f8d-8e52-112531e29105-kube-api-access-kljg6\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.571204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.571173    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:46.615072 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.611791    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be46f62a-1a48-4701-bac1-523b4398f80c-kube-api-access-bzrjz\" (UniqueName: \"kubernetes.io/projected/be46f62a-1a48-4701-bac1-523b4398f80c-kube-api-access-bzrjz\") pod \"be46f62a-1a48-4701-bac1-523b4398f80c\" (UID: \"be46f62a-1a48-4701-bac1-523b4398f80c\") "
Nov 02 02:09:46.621995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.621877    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:46.624154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.624126    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be46f62a-1a48-4701-bac1-523b4398f80c-kube-api-access-bzrjz" pod "be46f62a-1a48-4701-bac1-523b4398f80c" (UID: "be46f62a-1a48-4701-bac1-523b4398f80c"). InnerVolumeSpecName "kube-api-access-bzrjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:46.713816 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.713777    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzrjz\" (UniqueName: \"kubernetes.io/projected/be46f62a-1a48-4701-bac1-523b4398f80c-kube-api-access-bzrjz\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:46.803152 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.803030    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:46.916145 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.916110    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:46.918742 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:46.918723    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls\") pod \"maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:47.129475 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.129439    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:47.226432 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.226404    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be46f62a-1a48-4701-bac1-523b4398f80c" path="/var/lib/kubelet/pods/be46f62a-1a48-4701-bac1-523b4398f80c/volumes"
Nov 02 02:09:47.226784 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.226773    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc0224f-76d8-4279-8739-f6165803bd86" path="/var/lib/kubelet/pods/efc0224f-76d8-4279-8739-f6165803bd86/volumes"
Nov 02 02:09:47.321293 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.321261    4912 generic.go:356] "Generic (PLEG): container finished" podID="be46f62a-1a48-4701-bac1-523b4398f80c" containerID="60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d" exitCode=137
Nov 02 02:09:47.321449 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.321306    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-ba556b"
Nov 02 02:09:47.321449 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.321357    4912 scope.go:122] "RemoveContainer" containerID="60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d"
Nov 02 02:09:47.339144 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.339113    4912 scope.go:122] "RemoveContainer" containerID="60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d"
Nov 02 02:09:47.339605 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:47.339582    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d\": container with ID starting with 60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d not found: ID does not exist" containerID="60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d"
Nov 02 02:09:47.339693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.339617    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d"} err="failed to get container status \"60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d\": rpc error: code = NotFound desc = could not find container \"60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d\": container with ID starting with 60c3b3de8c72dbce64b167b25f9e6a75b578c38dd04023ba42a369285e70363d not found: ID does not exist"
Nov 02 02:09:47.483165 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.483132    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:47.522113 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.522041    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-xks-ca\") pod \"8729baf1-9239-482b-807e-1d1c23ac7b51\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") "
Nov 02 02:09:47.522406 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.522383    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-openshift-ca\") pod \"8729baf1-9239-482b-807e-1d1c23ac7b51\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") "
Nov 02 02:09:47.522649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.522631    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8729baf1-9239-482b-807e-1d1c23ac7b51-kube-api-access-n959x\" (UniqueName: \"kubernetes.io/projected/8729baf1-9239-482b-807e-1d1c23ac7b51-kube-api-access-n959x\") pod \"8729baf1-9239-482b-807e-1d1c23ac7b51\" (UID: \"8729baf1-9239-482b-807e-1d1c23ac7b51\") "
Nov 02 02:09:47.522776 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.522749    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-openshift-ca" pod "8729baf1-9239-482b-807e-1d1c23ac7b51" (UID: "8729baf1-9239-482b-807e-1d1c23ac7b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:47.523385 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.523346    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:47.528691 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.528665    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-xks-ca" pod "8729baf1-9239-482b-807e-1d1c23ac7b51" (UID: "8729baf1-9239-482b-807e-1d1c23ac7b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:47.528780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.528741    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8729baf1-9239-482b-807e-1d1c23ac7b51-kube-api-access-n959x" pod "8729baf1-9239-482b-807e-1d1c23ac7b51" (UID: "8729baf1-9239-482b-807e-1d1c23ac7b51"). InnerVolumeSpecName "kube-api-access-n959x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:47.571132 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.571100    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"]
Nov 02 02:09:47.571570 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:47.571542    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ce6436_0b1f_4f8d_8e52_112531e29105.slice/crio-e3d3088e1c615466e13143364926e3256ff96acca47dcad6bbad4fe40e2a848c WatchSource:0}: Error finding container e3d3088e1c615466e13143364926e3256ff96acca47dcad6bbad4fe40e2a848c: Status 404 returned error can't find the container with id e3d3088e1c615466e13143364926e3256ff96acca47dcad6bbad4fe40e2a848c
Nov 02 02:09:47.624414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.624378    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n959x\" (UniqueName: \"kubernetes.io/projected/8729baf1-9239-482b-807e-1d1c23ac7b51-kube-api-access-n959x\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:47.624414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:47.624415    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8729baf1-9239-482b-807e-1d1c23ac7b51-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:48.333373 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:48.333332    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"
Nov 02 02:09:48.333922 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:48.333374    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh" event={"ID":"8729baf1-9239-482b-807e-1d1c23ac7b51","Type":"ContainerDied","Data":"8912eb42c64ebb224ed02f6b0b60541210cf92252df9654e466a5865f2cdab0e"}
Nov 02 02:09:48.333922 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:48.333420    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8912eb42c64ebb224ed02f6b0b60541210cf92252df9654e466a5865f2cdab0e"
Nov 02 02:09:48.334793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:48.334770    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh" event={"ID":"d1ce6436-0b1f-4f8d-8e52-112531e29105","Type":"ContainerStarted","Data":"e3d3088e1c615466e13143364926e3256ff96acca47dcad6bbad4fe40e2a848c"}
Nov 02 02:09:49.353671 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.350306    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh" event={"ID":"d1ce6436-0b1f-4f8d-8e52-112531e29105","Type":"ContainerStarted","Data":"daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10"}
Nov 02 02:09:49.353671 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.350388    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:49.411754 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.411712    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:49.609182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.609036    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:49.794501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.794437    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh" podStartSLOduration=2.911237731 podStartE2EDuration="3.794417092s" totalImagesPullingTime="883.179361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:49.484146431 +0000 UTC m=+2282.389611788" watchObservedRunningTime="2026-11-02 02:09:49.794417092 +0000 UTC m=+2282.699882452"
Nov 02 02:09:49.795266 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.795243    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"]
Nov 02 02:09:49.796298 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.796276    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8729baf1-9239-482b-807e-1d1c23ac7b51" containerName="revoke-keys"
Nov 02 02:09:49.796298 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.796298    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8729baf1-9239-482b-807e-1d1c23ac7b51" containerName="revoke-keys"
Nov 02 02:09:49.796482 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.796468    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8729baf1-9239-482b-807e-1d1c23ac7b51" containerName="revoke-keys"
Nov 02 02:09:49.801369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.801338    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:49.802359 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.802338    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.851418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.850990    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.851418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.851048    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.851418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.851228    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29s7w\" (UniqueName: \"kubernetes.io/projected/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-kube-api-access-29s7w\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.851418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.851267    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.851418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.851324    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.852166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.851837    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.852166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.851886    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.852166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.851958    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.852166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.852007    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.884622 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.884589    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"]
Nov 02 02:09:49.953704 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.953670    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29s7w\" (UniqueName: \"kubernetes.io/projected/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-kube-api-access-29s7w\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.953704 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.953708    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.953976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.953751    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.953976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.953777    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.953976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.953795    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.953976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.953857    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.953976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.953906    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.953976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.953954    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.954363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.953993    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.954363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.954264    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.954731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.954705    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.955253 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.955226    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.955418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.955398    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.955525 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.955499    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.956587 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.956559    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:49.957294 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:49.957274    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:50.014671 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:50.014638    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:50.014859 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:50.014838    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29s7w\" (UniqueName: \"kubernetes.io/projected/1fa36757-ccb1-4e9d-af8b-2a7ad54b7178-kube-api-access-29s7w\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c7sz5\" (UID: \"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:50.117350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:50.117321    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:50.556843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:50.556817    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"]
Nov 02 02:09:50.559370 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:50.559341    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa36757_ccb1_4e9d_af8b_2a7ad54b7178.slice/crio-11053f3680bcf86c83e2f41cf1f434359bfb79aeeae8b6693b918c6ce832c21c WatchSource:0}: Error finding container 11053f3680bcf86c83e2f41cf1f434359bfb79aeeae8b6693b918c6ce832c21c: Status 404 returned error can't find the container with id 11053f3680bcf86c83e2f41cf1f434359bfb79aeeae8b6693b918c6ce832c21c
Nov 02 02:09:50.562403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:50.562364    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:09:50.562495 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:50.562454    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:09:50.562532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:50.562497    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:09:50.875506 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:50.875463    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:51.310452 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:51.310357    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:51.398272 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:51.397977    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" event={"ID":"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178","Type":"ContainerStarted","Data":"216242560e7cc49dc1d22cf0417a1319374b941319b67bd993bd156842ee2da9"}
Nov 02 02:09:51.398646 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:51.398601    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" event={"ID":"1fa36757-ccb1-4e9d-af8b-2a7ad54b7178","Type":"ContainerStarted","Data":"11053f3680bcf86c83e2f41cf1f434359bfb79aeeae8b6693b918c6ce832c21c"}
Nov 02 02:09:51.590017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:51.589889    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" podStartSLOduration=2.589864657 podStartE2EDuration="2.589864657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:51.579091226 +0000 UTC m=+2284.484556582" watchObservedRunningTime="2026-11-02 02:09:51.589864657 +0000 UTC m=+2284.495330015"
Nov 02 02:09:52.119412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:52.119370    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:52.121399 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:52.121368    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c7sz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 02:09:52.121517 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:52.121420    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" podUID="1fa36757-ccb1-4e9d-af8b-2a7ad54b7178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 02:09:53.117726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:53.117694    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c7sz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 02:09:53.118202 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:53.117746    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" podUID="1fa36757-ccb1-4e9d-af8b-2a7ad54b7178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 02:09:53.217089 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:53.217020    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:53.541713 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:53.541627    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:53.715754 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:53.715720    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:53.775084 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:53.772043    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"]
Nov 02 02:09:53.805833 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:53.805762    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-55bc84-p6xw8"]
Nov 02 02:09:54.117634 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.117600    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c7sz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 02:09:54.117853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.117661    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" podUID="1fa36757-ccb1-4e9d-af8b-2a7ad54b7178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 02:09:54.175995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.175947    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:54.379188 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.379099    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"]
Nov 02 02:09:54.384440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.384414    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.442493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.442445    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"]
Nov 02 02:09:54.476122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.468490    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.476122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.468552    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.476122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.468621    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19ab8da-5870-43b6-be66-998a9c154b93-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.476122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.468668    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.476122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.468738    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.476122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.468762    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19ab8da-5870-43b6-be66-998a9c154b93-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.476122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.468844    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87nb4\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-kube-api-access-87nb4\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.476122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.468875    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.476122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.468933    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.483603 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.483577    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5700d3d1-openshift-default-dockercfg-z5s4p\""
Nov 02 02:09:54.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.570795    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19ab8da-5870-43b6-be66-998a9c154b93-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.570878    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.570953    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.570979    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19ab8da-5870-43b6-be66-998a9c154b93-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.571082    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87nb4\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-kube-api-access-87nb4\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.571113    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.571152    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.571249    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.572079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.571284    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.577703 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.577672    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19ab8da-5870-43b6-be66-998a9c154b93-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.577900 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.577875    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.578155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.578128    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.578424 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.578401    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.578547 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.578463    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.581228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.581202    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.581526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.581504    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19ab8da-5870-43b6-be66-998a9c154b93-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.637772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.637699    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.637918 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.637878    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87nb4\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-kube-api-access-87nb4\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:54.699103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:54.699069    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:55.117865 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.117835    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c7sz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 02:09:55.118278 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.117888    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" podUID="1fa36757-ccb1-4e9d-af8b-2a7ad54b7178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 02:09:55.177640 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:09:55.177610    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19ab8da_5870_43b6_be66_998a9c154b93.slice/crio-595ef76046092bd56b33abde93e25ceca6e5beb9e1f056de8205c5da1f90a944 WatchSource:0}: Error finding container 595ef76046092bd56b33abde93e25ceca6e5beb9e1f056de8205c5da1f90a944: Status 404 returned error can't find the container with id 595ef76046092bd56b33abde93e25ceca6e5beb9e1f056de8205c5da1f90a944
Nov 02 02:09:55.180400 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.180369    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:09:55.180522 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.180447    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:09:55.180522 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.180494    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:09:55.182113 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.182088    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"]
Nov 02 02:09:55.231557 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.231533    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17299572-06c7-4ac0-9bde-2cc8ac874cb5" path="/var/lib/kubelet/pods/17299572-06c7-4ac0-9bde-2cc8ac874cb5/volumes"
Nov 02 02:09:55.378178 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.377280    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:09:55.431998 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.431960    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" event={"ID":"b19ab8da-5870-43b6-be66-998a9c154b93","Type":"ContainerStarted","Data":"e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0"}
Nov 02 02:09:55.432203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.432016    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" event={"ID":"b19ab8da-5870-43b6-be66-998a9c154b93","Type":"ContainerStarted","Data":"595ef76046092bd56b33abde93e25ceca6e5beb9e1f056de8205c5da1f90a944"}
Nov 02 02:09:55.680542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.680424    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" podStartSLOduration=1.6804037630000002 podStartE2EDuration="1.680403763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:55.658496572 +0000 UTC m=+2288.563961930" watchObservedRunningTime="2026-11-02 02:09:55.680403763 +0000 UTC m=+2288.585869187"
Nov 02 02:09:55.700140 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:55.700113    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:09:56.098903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:56.098787    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:56.118624 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:56.118589    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c7sz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 02:09:56.119012 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:56.118639    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" podUID="1fa36757-ccb1-4e9d-af8b-2a7ad54b7178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 02:09:56.703106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:56.702468    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 02:09:56.703106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:56.702537    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 02:09:57.095419 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:57.095298    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:57.117964 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:57.117840    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c7sz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 02:09:57.117964 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:57.117914    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" podUID="1fa36757-ccb1-4e9d-af8b-2a7ad54b7178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 02:09:57.278894 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:57.278855    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:57.699767 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:57.699723    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 02:09:57.699992 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:57.699787    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 02:09:58.118633 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:58.118595    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c7sz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 02:09:58.118840 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:58.118684    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5" podUID="1fa36757-ccb1-4e9d-af8b-2a7ad54b7178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 02:09:58.699995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:58.699939    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 02:09:58.700601 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:58.700015    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 02:09:59.597668 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.597623    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"]
Nov 02 02:09:59.603543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.603514    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.668084 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.667330    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4211c0-serving-cert\""
Nov 02 02:09:59.700464 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.700416    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 02:09:59.700948 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.700479    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 02:09:59.704508 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.704471    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-metrics-certs\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.704685 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.704602    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.704814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.704791    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qplsk\" (UniqueName: \"kubernetes.io/projected/cc459d6e-0d38-4103-959c-d11466ca4d0d-kube-api-access-qplsk\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.710132 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.710097    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc459d6e-0d38-4103-959c-d11466ca4d0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.723985 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.723931    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"]
Nov 02 02:09:59.816926 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.816886    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc459d6e-0d38-4103-959c-d11466ca4d0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.819143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.817737    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc459d6e-0d38-4103-959c-d11466ca4d0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.819143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.818006    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-metrics-certs\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.819143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.818197    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.819143 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:59.818311    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-serving-cert: secret "maas-api-e2e-inf-b-4211c0-serving-cert" not found
Nov 02 02:09:59.819143 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:09:59.818371    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls podName:cc459d6e-0d38-4103-959c-d11466ca4d0d nodeName:}" failed. No retries permitted until 2026-11-02 02:10:00.31834936 +0000 UTC m=+2293.223814714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls") pod "maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl" (UID: "cc459d6e-0d38-4103-959c-d11466ca4d0d") : secret "maas-api-e2e-inf-b-4211c0-serving-cert" not found
Nov 02 02:09:59.819143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.818533    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qplsk\" (UniqueName: \"kubernetes.io/projected/cc459d6e-0d38-4103-959c-d11466ca4d0d-kube-api-access-qplsk\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.822748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.822653    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-metrics-certs\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:09:59.847736 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.847523    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:09:59.901155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.872441    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:59.901155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.873970    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:59.901155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.899389    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c7sz5"
Nov 02 02:09:59.906799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:09:59.906768    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qplsk\" (UniqueName: \"kubernetes.io/projected/cc459d6e-0d38-4103-959c-d11466ca4d0d-kube-api-access-qplsk\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:10:00.035403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.035363    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:00.215089 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.212761    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:00.326449 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.326405    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:10:00.329335 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.329308    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:10:00.521407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.521326    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:10:00.675209 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.671274    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:00.700995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.700959    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 02:10:00.701418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.701021    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 02:10:00.874729 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.874693    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:00.922369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:00.922332    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"]
Nov 02 02:10:00.922760 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:10:00.922729    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc459d6e_0d38_4103_959c_d11466ca4d0d.slice/crio-489859be08217a175fd7c9f1fe23a3d1930104cb5a76978884160c3929079129 WatchSource:0}: Error finding container 489859be08217a175fd7c9f1fe23a3d1930104cb5a76978884160c3929079129: Status 404 returned error can't find the container with id 489859be08217a175fd7c9f1fe23a3d1930104cb5a76978884160c3929079129
Nov 02 02:10:01.019831 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:01.019777    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"]
Nov 02 02:10:01.021597 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:01.021556    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy" containerID="cri-o://ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c" gracePeriod=30
Nov 02 02:10:01.086174 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:01.086134    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:01.524606 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:01.524566    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl" event={"ID":"cc459d6e-0d38-4103-959c-d11466ca4d0d","Type":"ContainerStarted","Data":"489859be08217a175fd7c9f1fe23a3d1930104cb5a76978884160c3929079129"}
Nov 02 02:10:02.302049 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:02.302019    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:10:02.302575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:02.302379    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:10:02.303312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:02.303295    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:10:02.535136 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:02.535091    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl" event={"ID":"cc459d6e-0d38-4103-959c-d11466ca4d0d","Type":"ContainerStarted","Data":"b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b"}
Nov 02 02:10:02.535332 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:02.535312    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:10:02.666143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:02.666089    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl" podStartSLOduration=2.795710979 podStartE2EDuration="3.666070351s" totalImagesPullingTime="870.359372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:10:02.659333785 +0000 UTC m=+2295.564799142" watchObservedRunningTime="2026-11-02 02:10:02.666070351 +0000 UTC m=+2295.571535707"
Nov 02 02:10:05.539854 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:05.539811    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:06.305385 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.305306    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 02:10:06.406473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406439    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-kube-api-access-dl4mq\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-kube-api-access-dl4mq\") pod \"44774b42-edee-4b78-b80c-70272d3af170\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") "
Nov 02 02:10:06.406694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406488    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-socket\") pod \"44774b42-edee-4b78-b80c-70272d3af170\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") "
Nov 02 02:10:06.406694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406551    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-envoy\") pod \"44774b42-edee-4b78-b80c-70272d3af170\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") "
Nov 02 02:10:06.406694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406569    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-istio-token\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-istio-token\") pod \"44774b42-edee-4b78-b80c-70272d3af170\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") "
Nov 02 02:10:06.406694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406661    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44774b42-edee-4b78-b80c-70272d3af170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44774b42-edee-4b78-b80c-70272d3af170-istio-podinfo\") pod \"44774b42-edee-4b78-b80c-70272d3af170\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") "
Nov 02 02:10:06.406930 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406735    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-certs\") pod \"44774b42-edee-4b78-b80c-70272d3af170\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") "
Nov 02 02:10:06.406930 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406762    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-data\") pod \"44774b42-edee-4b78-b80c-70272d3af170\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") "
Nov 02 02:10:06.406930 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406793    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-credential-socket\") pod \"44774b42-edee-4b78-b80c-70272d3af170\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") "
Nov 02 02:10:06.406930 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406862    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44774b42-edee-4b78-b80c-70272d3af170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44774b42-edee-4b78-b80c-70272d3af170-istiod-ca-cert\") pod \"44774b42-edee-4b78-b80c-70272d3af170\" (UID: \"44774b42-edee-4b78-b80c-70272d3af170\") "
Nov 02 02:10:06.406930 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.406894    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-socket" pod "44774b42-edee-4b78-b80c-70272d3af170" (UID: "44774b42-edee-4b78-b80c-70272d3af170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.407253 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.407141    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-certs" pod "44774b42-edee-4b78-b80c-70272d3af170" (UID: "44774b42-edee-4b78-b80c-70272d3af170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.407613 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.407586    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-data" pod "44774b42-edee-4b78-b80c-70272d3af170" (UID: "44774b42-edee-4b78-b80c-70272d3af170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.407823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.407646    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44774b42-edee-4b78-b80c-70272d3af170-istiod-ca-cert" pod "44774b42-edee-4b78-b80c-70272d3af170" (UID: "44774b42-edee-4b78-b80c-70272d3af170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:06.407921 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.407759    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-credential-socket" pod "44774b42-edee-4b78-b80c-70272d3af170" (UID: "44774b42-edee-4b78-b80c-70272d3af170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.407921 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.407767    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.407921 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.407866    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.409104 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.409048    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-kube-api-access-dl4mq" pod "44774b42-edee-4b78-b80c-70272d3af170" (UID: "44774b42-edee-4b78-b80c-70272d3af170"). InnerVolumeSpecName "kube-api-access-dl4mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:06.409283 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.409266    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-envoy" pod "44774b42-edee-4b78-b80c-70272d3af170" (UID: "44774b42-edee-4b78-b80c-70272d3af170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.409940 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.409916    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44774b42-edee-4b78-b80c-70272d3af170-istio-podinfo" pod "44774b42-edee-4b78-b80c-70272d3af170" (UID: "44774b42-edee-4b78-b80c-70272d3af170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:10:06.410041 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.409916    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-istio-token" pod "44774b42-edee-4b78-b80c-70272d3af170" (UID: "44774b42-edee-4b78-b80c-70272d3af170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:06.508896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.508846    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.508896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.508889    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.508896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.508904    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44774b42-edee-4b78-b80c-70272d3af170-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.509172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.508918    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.509172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.508930    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44774b42-edee-4b78-b80c-70272d3af170-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.509172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.508941    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44774b42-edee-4b78-b80c-70272d3af170-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.509172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.508952    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4mq\" (UniqueName: \"kubernetes.io/projected/44774b42-edee-4b78-b80c-70272d3af170-kube-api-access-dl4mq\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.576321 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.576238    4912 generic.go:356] "Generic (PLEG): container finished" podID="44774b42-edee-4b78-b80c-70272d3af170" containerID="ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c" exitCode=0
Nov 02 02:10:06.576321 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.576295    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"
Nov 02 02:10:06.576786 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.576326    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" event={"ID":"44774b42-edee-4b78-b80c-70272d3af170","Type":"ContainerDied","Data":"ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c"}
Nov 02 02:10:06.576786 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.576389    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh" event={"ID":"44774b42-edee-4b78-b80c-70272d3af170","Type":"ContainerDied","Data":"b4f127efdb55a1be1a09afed72ccbb45bc1e35223e2d598846a76e55e875b2d2"}
Nov 02 02:10:06.576786 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.576416    4912 scope.go:122] "RemoveContainer" containerID="ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c"
Nov 02 02:10:06.590444 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.590422    4912 scope.go:122] "RemoveContainer" containerID="3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c"
Nov 02 02:10:06.602701 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.602678    4912 scope.go:122] "RemoveContainer" containerID="ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c"
Nov 02 02:10:06.603023 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:10:06.603000    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c\": container with ID starting with ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c not found: ID does not exist" containerID="ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c"
Nov 02 02:10:06.603175 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.603033    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c"} err="failed to get container status \"ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c\": rpc error: code = NotFound desc = could not find container \"ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c\": container with ID starting with ed50a5eaa472721cfecb2f65d1390ea21c039f25dfddec97bfca139cafaaa34c not found: ID does not exist"
Nov 02 02:10:06.603175 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.603079    4912 scope.go:122] "RemoveContainer" containerID="3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c"
Nov 02 02:10:06.603374 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:10:06.603355    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c\": container with ID starting with 3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c not found: ID does not exist" containerID="3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c"
Nov 02 02:10:06.603450 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.603381    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c"} err="failed to get container status \"3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c\": rpc error: code = NotFound desc = could not find container \"3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c\": container with ID starting with 3bb445532eec31e5620490d3cce29145b5152aa953c9ba00e062e75a3eff902c not found: ID does not exist"
Nov 02 02:10:06.743071 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.743026    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"]
Nov 02 02:10:06.748319 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:06.748294    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tbpfh"]
Nov 02 02:10:07.225762 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:07.225727    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44774b42-edee-4b78-b80c-70272d3af170" path="/var/lib/kubelet/pods/44774b42-edee-4b78-b80c-70272d3af170/volumes"
Nov 02 02:10:08.549541 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:08.549513    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:10:09.492034 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.492000    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"]
Nov 02 02:10:09.492725 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.492706    4912 cpu_manager.go:414] "containerMap: removing container" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy"
Nov 02 02:10:09.492823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.492727    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy"
Nov 02 02:10:09.492887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.492850    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy"
Nov 02 02:10:09.492887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.492863    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy"
Nov 02 02:10:09.498025 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.498004    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.541890 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.541866    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"]
Nov 02 02:10:09.542442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.542425    4912 cpu_manager.go:414] "containerMap: removing container" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy"
Nov 02 02:10:09.542507 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.542444    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44774b42-edee-4b78-b80c-70272d3af170" containerName="istio-proxy"
Nov 02 02:10:09.546250 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.546230    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.600189 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.600155    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"]
Nov 02 02:10:09.645602 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.645566    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.645826 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.645614    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.645826 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.645638    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzl5p\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-kube-api-access-pzl5p\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.645977 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.645849    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6995d05c-1099-418f-9905-c65248d5c660-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.645977 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.645903    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.646132 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646006    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.646132 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646083    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.646250 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646187    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.646250 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646215    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.646329 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646266    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4zs\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-kube-api-access-ms4zs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.646329 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646305    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.646329 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646326    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6995d05c-1099-418f-9905-c65248d5c660-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.646489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646354    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.646489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646374    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.646489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646439    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.646586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646527    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.646586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646549    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.646586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.646567    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.680112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.680034    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"]
Nov 02 02:10:09.747459 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747377    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.747459 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747424    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.747459 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747450    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzl5p\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-kube-api-access-pzl5p\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.747756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747535    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6995d05c-1099-418f-9905-c65248d5c660-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.747756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747569    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.747756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747602    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.747756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747645    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.747756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747728    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.747756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747755    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.748098 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747779    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4zs\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-kube-api-access-ms4zs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.748098 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747817    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.748098 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747848    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6995d05c-1099-418f-9905-c65248d5c660-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.748098 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747895    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.748098 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747904    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.748098 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747963    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.748440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.747900    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.748440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748187    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.748440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748208    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.748440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748237    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.748440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748275    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.748440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748280    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.748440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748304    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.748440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748334    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.749076 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748639    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.749076 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748855    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6995d05c-1099-418f-9905-c65248d5c660-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.749076 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.748881    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.749253 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.749146    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.749379 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.749350    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.750593 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.750548    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.750848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.750824    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.752023 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.752002    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6995d05c-1099-418f-9905-c65248d5c660-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.752379 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.752358    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.809339 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.809308    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4zs\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-kube-api-access-ms4zs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.809772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.809752    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.809876 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.809853    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.810389 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.810363    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzl5p\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-kube-api-access-pzl5p\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:09.811902 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.811880    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:09.860495 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:09.860405    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:10.274935 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.274903    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"]
Nov 02 02:10:10.275325 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:10:10.275296    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c0dd23_fea9_49e6_af8c_4f67d31d4eb4.slice/crio-515a6cb4d8cc02202c836513b255a5f23f2aaab7c7bb19f1c0d45594d58e38f2 WatchSource:0}: Error finding container 515a6cb4d8cc02202c836513b255a5f23f2aaab7c7bb19f1c0d45594d58e38f2: Status 404 returned error can't find the container with id 515a6cb4d8cc02202c836513b255a5f23f2aaab7c7bb19f1c0d45594d58e38f2
Nov 02 02:10:10.277808 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.277773    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:10:10.277939 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.277842    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:10:10.277939 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.277881    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:10:10.324960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.324935    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"]
Nov 02 02:10:10.325671 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:10:10.325643    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6995d05c_1099_418f_9905_c65248d5c660.slice/crio-b84de789a812f020736ab8f5d8f3a0e75e146f996972b23831edfdd79d622d3c WatchSource:0}: Error finding container b84de789a812f020736ab8f5d8f3a0e75e146f996972b23831edfdd79d622d3c: Status 404 returned error can't find the container with id b84de789a812f020736ab8f5d8f3a0e75e146f996972b23831edfdd79d622d3c
Nov 02 02:10:10.328443 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.328411    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:10:10.328534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.328479    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:10:10.328534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.328512    4912 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 02:10:10.344235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.344209    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:10.630112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.630070    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" event={"ID":"6995d05c-1099-418f-9905-c65248d5c660","Type":"ContainerStarted","Data":"0c02136be1ecd6c5dd2ce10fbe70974b4e41edea483839e8363769071d161881"}
Nov 02 02:10:10.630839 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.630137    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" event={"ID":"6995d05c-1099-418f-9905-c65248d5c660","Type":"ContainerStarted","Data":"b84de789a812f020736ab8f5d8f3a0e75e146f996972b23831edfdd79d622d3c"}
Nov 02 02:10:10.632324 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.632289    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" event={"ID":"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4","Type":"ContainerStarted","Data":"a0c2701b6ca1bb6293c0385e95ffc654ab995fbb141404197614163db611cf5f"}
Nov 02 02:10:10.632474 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.632351    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" event={"ID":"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4","Type":"ContainerStarted","Data":"515a6cb4d8cc02202c836513b255a5f23f2aaab7c7bb19f1c0d45594d58e38f2"}
Nov 02 02:10:10.756105 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.750221    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" podStartSLOduration=1.750202871 podStartE2EDuration="1.750202871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:10:10.749304885 +0000 UTC m=+2303.654770280" watchObservedRunningTime="2026-11-02 02:10:10.750202871 +0000 UTC m=+2303.655668232"
Nov 02 02:10:10.812191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.812163    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:10.860885 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.860850    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:10.878482 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:10.878425    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" podStartSLOduration=1.878410849 podStartE2EDuration="1.878410849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:10:10.873577106 +0000 UTC m=+2303.779042465" watchObservedRunningTime="2026-11-02 02:10:10.878410849 +0000 UTC m=+2303.783876205"
Nov 02 02:10:11.813875 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:11.813833    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 02:10:11.814392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:11.813898    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 02:10:11.884217 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:11.884178    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 02:10:11.885162 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:11.885125    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 02:10:12.143248 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:10:12.143215    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:10:12.812418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:12.812380    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 02:10:12.812707 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:12.812680    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 02:10:12.863462 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:12.863425    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 02:10:12.864082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:12.864029    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 02:10:13.813224 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:13.812486    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 02:10:13.813224 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:13.812568    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 02:10:13.862206 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:13.861436    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 02:10:13.862206 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:13.861531    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 02:10:14.812266 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:14.812188    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 02:10:14.812766 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:14.812274    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 02:10:14.868003 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:14.867966    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 02:10:14.868322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:14.868290    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 02:10:15.124944 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:15.124906    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:15.812389 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:15.812343    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 02:10:15.812907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:15.812401    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 02:10:15.861372 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:15.861336    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 02:10:15.861675 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:15.861647    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 02:10:16.813127 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:16.813082    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 02:10:16.813776 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:16.813160    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 02:10:16.870083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:16.867586    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 02:10:16.870083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:16.867647    4912 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 02:10:17.954550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:17.952012    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:17.954550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:17.953391    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:17.955177 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:17.954917    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:10:17.967456 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:17.967427    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:17.970097 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:17.968988    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:17.970097 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:17.969403    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:10:19.398825 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:19.398778    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:20.618375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:20.618336    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:21.019672 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:21.019577    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:21.329983 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:21.329896    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:21.660479 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:21.660440    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"]
Nov 02 02:10:21.709698 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:21.709666    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-061fab-8k7s6"]
Nov 02 02:10:21.988977 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:21.988886    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:22.484154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:22.484117    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:22.820835 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:22.820730    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:23.001475 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:23.001443    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:23.048465 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:23.048434    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"]
Nov 02 02:10:23.084344 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:23.084258    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5700d3d1-rdgwh"]
Nov 02 02:10:23.226560 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:23.226531    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1c0f26-3660-4874-a458-402952ce2612" path="/var/lib/kubelet/pods/0c1c0f26-3660-4874-a458-402952ce2612/volumes"
Nov 02 02:10:23.227021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:23.227008    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8729baf1-9239-482b-807e-1d1c23ac7b51" path="/var/lib/kubelet/pods/8729baf1-9239-482b-807e-1d1c23ac7b51/volumes"
Nov 02 02:10:23.505763 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:23.505727    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:27.957117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:27.957048    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:28.140161 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:28.140125    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:28.265022 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:28.264935    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"]
Nov 02 02:10:28.265260 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:28.265181    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy" containerID="cri-o://c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4" gracePeriod=30
Nov 02 02:10:28.314251 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:28.314217    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:28.549603 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:28.549512    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:30.480333 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:30.480287    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:30.684427 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:30.684384    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:32.505314 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:32.505278    4912 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:10:32.505932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:32.505898    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:10:33.477019 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.476924    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:33.559248 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.559222    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 02:10:33.665998 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.665965    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-data\") pod \"d2d28689-caf9-4713-b737-c581416b28a9\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") "
Nov 02 02:10:33.666213 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666008    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-certs\") pod \"d2d28689-caf9-4713-b737-c581416b28a9\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") "
Nov 02 02:10:33.666213 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666095    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-kube-api-access-gp58j\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-kube-api-access-gp58j\") pod \"d2d28689-caf9-4713-b737-c581416b28a9\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") "
Nov 02 02:10:33.666213 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666152    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-credential-socket\") pod \"d2d28689-caf9-4713-b737-c581416b28a9\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") "
Nov 02 02:10:33.666213 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666179    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-socket\") pod \"d2d28689-caf9-4713-b737-c581416b28a9\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") "
Nov 02 02:10:33.666470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666267    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2d28689-caf9-4713-b737-c581416b28a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d28689-caf9-4713-b737-c581416b28a9-istio-podinfo\") pod \"d2d28689-caf9-4713-b737-c581416b28a9\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") "
Nov 02 02:10:33.666470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666302    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-envoy\") pod \"d2d28689-caf9-4713-b737-c581416b28a9\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") "
Nov 02 02:10:33.666470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666325    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-istio-token\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-istio-token\") pod \"d2d28689-caf9-4713-b737-c581416b28a9\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") "
Nov 02 02:10:33.666470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666389    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d28689-caf9-4713-b737-c581416b28a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d28689-caf9-4713-b737-c581416b28a9-istiod-ca-cert\") pod \"d2d28689-caf9-4713-b737-c581416b28a9\" (UID: \"d2d28689-caf9-4713-b737-c581416b28a9\") "
Nov 02 02:10:33.666470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666316    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-data" pod "d2d28689-caf9-4713-b737-c581416b28a9" (UID: "d2d28689-caf9-4713-b737-c581416b28a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:33.666470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666334    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-certs" pod "d2d28689-caf9-4713-b737-c581416b28a9" (UID: "d2d28689-caf9-4713-b737-c581416b28a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:33.666778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666464    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-credential-socket" pod "d2d28689-caf9-4713-b737-c581416b28a9" (UID: "d2d28689-caf9-4713-b737-c581416b28a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:33.666778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666503    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-socket" pod "d2d28689-caf9-4713-b737-c581416b28a9" (UID: "d2d28689-caf9-4713-b737-c581416b28a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:33.666893 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666849    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d28689-caf9-4713-b737-c581416b28a9-istiod-ca-cert" pod "d2d28689-caf9-4713-b737-c581416b28a9" (UID: "d2d28689-caf9-4713-b737-c581416b28a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:33.667191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.666977    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:33.667191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.667086    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:33.667191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.667140    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:33.667191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.667165    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:33.668751 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.668722    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-kube-api-access-gp58j" pod "d2d28689-caf9-4713-b737-c581416b28a9" (UID: "d2d28689-caf9-4713-b737-c581416b28a9"). InnerVolumeSpecName "kube-api-access-gp58j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:33.668853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.668746    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-envoy" pod "d2d28689-caf9-4713-b737-c581416b28a9" (UID: "d2d28689-caf9-4713-b737-c581416b28a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:33.668853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.668764    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-istio-token" pod "d2d28689-caf9-4713-b737-c581416b28a9" (UID: "d2d28689-caf9-4713-b737-c581416b28a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:33.668853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.668801    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2d28689-caf9-4713-b737-c581416b28a9-istio-podinfo" pod "d2d28689-caf9-4713-b737-c581416b28a9" (UID: "d2d28689-caf9-4713-b737-c581416b28a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:10:33.768848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.768814    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d28689-caf9-4713-b737-c581416b28a9-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:33.768848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.768844    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d28689-caf9-4713-b737-c581416b28a9-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:33.768848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.768856    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:33.769247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.768864    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d28689-caf9-4713-b737-c581416b28a9-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:33.769247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.768875    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp58j\" (UniqueName: \"kubernetes.io/projected/d2d28689-caf9-4713-b737-c581416b28a9-kube-api-access-gp58j\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:33.909051 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.909017    4912 generic.go:356] "Generic (PLEG): container finished" podID="d2d28689-caf9-4713-b737-c581416b28a9" containerID="c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4" exitCode=0
Nov 02 02:10:33.909254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.909079    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"
Nov 02 02:10:33.909254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.909097    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" event={"ID":"d2d28689-caf9-4713-b737-c581416b28a9","Type":"ContainerDied","Data":"c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4"}
Nov 02 02:10:33.909254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.909146    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf" event={"ID":"d2d28689-caf9-4713-b737-c581416b28a9","Type":"ContainerDied","Data":"e90f97792a6029468cc99c3e2b245ed9f1536b7a05c8bf3ec8651c370f693963"}
Nov 02 02:10:33.909254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.909168    4912 scope.go:122] "RemoveContainer" containerID="c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4"
Nov 02 02:10:33.922176 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.922156    4912 scope.go:122] "RemoveContainer" containerID="c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4"
Nov 02 02:10:33.922522 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:10:33.922502    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4\": container with ID starting with c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4 not found: ID does not exist" containerID="c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4"
Nov 02 02:10:33.922582 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:33.922531    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4"} err="failed to get container status \"c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4\": rpc error: code = NotFound desc = could not find container \"c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4\": container with ID starting with c1d3d69b106c575ad7d075dcfeb192836d9403307ce4215517d5b9e09b3538f4 not found: ID does not exist"
Nov 02 02:10:34.074928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:34.074883    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"]
Nov 02 02:10:34.081104 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:34.081050    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-8tsqf"]
Nov 02 02:10:35.226666 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:35.226634    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d28689-caf9-4713-b737-c581416b28a9" path="/var/lib/kubelet/pods/d2d28689-caf9-4713-b737-c581416b28a9/volumes"
Nov 02 02:10:38.393169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:38.393136    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:42.137761 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:10:42.137721    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:10:42.955884 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:42.955847    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:43.470919 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:43.470882    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:43.775979 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:43.775877    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:44.364225 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:44.364185    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:44.597783 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:44.597750    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:48.607961 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:48.607916    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:49.013192 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:49.013148    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:49.199218 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:49.199126    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:49.383637 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:49.383594    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:51.225204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:51.225166    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:53.143401 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:53.143365    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:53.665991 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:53.659144    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:53.926804 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:53.926709    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:54.108234 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:54.108179    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:54.317417 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:54.317320    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:54.594413 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:54.594327    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:10:58.534326 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:10:58.534294    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:00.661952 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:00.661916    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:03.341029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:03.340990    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:05.495725 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:05.495669    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:05.885823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:05.885788    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:08.229755 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:08.229714    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:11.659122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:11.659040    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:12.138202 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:11:12.138159    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:11:13.022516 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:13.022475    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:17.958659 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:17.958618    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:20.883818 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.883776    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"]
Nov 02 02:11:20.884691 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.884426    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy"
Nov 02 02:11:20.884691 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.884440    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy"
Nov 02 02:11:20.884691 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.884538    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d28689-caf9-4713-b737-c581416b28a9" containerName="istio-proxy"
Nov 02 02:11:20.888103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.888084    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:20.947383 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.947349    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"]
Nov 02 02:11:20.990879 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.990824    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1c942d\"/\"kube-root-ca.crt\""
Nov 02 02:11:20.990879 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.990826    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1c942d\"/\"openshift-service-ca.crt\""
Nov 02 02:11:20.991245 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.991086    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1c942d\"/\"test-model-1c942d-kserve-self-signed-certs\""
Nov 02 02:11:20.991245 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:20.991182    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1c942d\"/\"default-dockercfg-xn8wd\""
Nov 02 02:11:21.007468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.007436    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:21.015970 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.015936    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwn7d\" (UniqueName: \"kubernetes.io/projected/1a4bb112-0081-4f0f-9f12-914574e2a13a-kube-api-access-cwn7d\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.016172 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.016097    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb112-0081-4f0f-9f12-914574e2a13a-tls-certs\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.016244 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.016226    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-tmp-dir\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.016302 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.016261    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-dshm\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.016352 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.016297    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-home\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.016405 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.016365    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-model-cache\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.117843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.117807    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb112-0081-4f0f-9f12-914574e2a13a-tls-certs\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.118088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.117900    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-tmp-dir\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.118088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.117923    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-dshm\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.118088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.117939    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-home\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.118088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.117978    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-model-cache\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.118294 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.118163    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwn7d\" (UniqueName: \"kubernetes.io/projected/1a4bb112-0081-4f0f-9f12-914574e2a13a-kube-api-access-cwn7d\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.118583 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.118554    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-tmp-dir\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.118695 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.118593    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-home\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.118791 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.118711    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-model-cache\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.120527 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.120506    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-dshm\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.120975 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.120960    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb112-0081-4f0f-9f12-914574e2a13a-tls-certs\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.180843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.180769    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwn7d\" (UniqueName: \"kubernetes.io/projected/1a4bb112-0081-4f0f-9f12-914574e2a13a-kube-api-access-cwn7d\") pod \"test-model-1c942d-kserve-85d795497-78585\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") " pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.201657 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.201623    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:21.592169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:21.592132    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"]
Nov 02 02:11:21.592730 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:11:21.592701    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a4bb112_0081_4f0f_9f12_914574e2a13a.slice/crio-f82691326a6841d40ed01c55d8aec7fbae4d59cd8be6fa789f4d09605cca6889 WatchSource:0}: Error finding container f82691326a6841d40ed01c55d8aec7fbae4d59cd8be6fa789f4d09605cca6889: Status 404 returned error can't find the container with id f82691326a6841d40ed01c55d8aec7fbae4d59cd8be6fa789f4d09605cca6889
Nov 02 02:11:22.383626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:22.383583    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585" event={"ID":"1a4bb112-0081-4f0f-9f12-914574e2a13a","Type":"ContainerStarted","Data":"6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34"}
Nov 02 02:11:22.383626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:22.383632    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585" event={"ID":"1a4bb112-0081-4f0f-9f12-914574e2a13a","Type":"ContainerStarted","Data":"f82691326a6841d40ed01c55d8aec7fbae4d59cd8be6fa789f4d09605cca6889"}
Nov 02 02:11:22.503661 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:22.503596    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585" podStartSLOduration=2.5035816669999997 podStartE2EDuration="2.503581667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:11:22.499781698 +0000 UTC m=+2375.405247056" watchObservedRunningTime="2026-11-02 02:11:22.503581667 +0000 UTC m=+2375.409047023"
Nov 02 02:11:22.746924 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:22.746843    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:23.202610 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:23.202576    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:25.694365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:25.694327    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:26.787299 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:26.787264    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:26.989411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:26.989370    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:27.138091 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.138039    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"]
Nov 02 02:11:27.138323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.138299    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy" containerID="cri-o://e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0" gracePeriod=30
Nov 02 02:11:27.155163 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.155127    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"]
Nov 02 02:11:27.155389 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.155364    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy" containerID="cri-o://a0c2701b6ca1bb6293c0385e95ffc654ab995fbb141404197614163db611cf5f" gracePeriod=30
Nov 02 02:11:27.170005 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.169974    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:27.176327 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.176305    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"]
Nov 02 02:11:27.176511 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.176493    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy" containerID="cri-o://0c02136be1ecd6c5dd2ce10fbe70974b4e41edea483839e8363769071d161881" gracePeriod=30
Nov 02 02:11:27.280854 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.280813    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"]
Nov 02 02:11:27.287604 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.287562    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.339199 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.339165    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"]
Nov 02 02:11:27.339829 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.339808    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-76837e-serving-cert\""
Nov 02 02:11:27.401348 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.401238    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-metrics-certs\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.401348 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.401329    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xq8v\" (UniqueName: \"kubernetes.io/projected/e50d6d24-b770-400d-bf1b-c3ceade8a35c-kube-api-access-6xq8v\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.401668 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.401432    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.401668 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.401464    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50d6d24-b770-400d-bf1b-c3ceade8a35c-openshift-service-ca\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.502538 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.502501    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-metrics-certs\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.502726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.502572    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xq8v\" (UniqueName: \"kubernetes.io/projected/e50d6d24-b770-400d-bf1b-c3ceade8a35c-kube-api-access-6xq8v\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.502726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.502627    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.502726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.502642    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50d6d24-b770-400d-bf1b-c3ceade8a35c-openshift-service-ca\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.502914 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:11:27.502795    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-serving-cert: secret "maas-api-e2e-api-b-76837e-serving-cert" not found
Nov 02 02:11:27.502914 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:11:27.502872    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls podName:e50d6d24-b770-400d-bf1b-c3ceade8a35c nodeName:}" failed. No retries permitted until 2026-11-02 02:11:28.002852873 +0000 UTC m=+2380.908318210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls") pod "maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk" (UID: "e50d6d24-b770-400d-bf1b-c3ceade8a35c") : secret "maas-api-e2e-api-b-76837e-serving-cert" not found
Nov 02 02:11:27.503452 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.503419    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:27.503589 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.503530    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50d6d24-b770-400d-bf1b-c3ceade8a35c-openshift-service-ca\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.505308 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.505283    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-metrics-certs\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.565734 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.565706    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xq8v\" (UniqueName: \"kubernetes.io/projected/e50d6d24-b770-400d-bf1b-c3ceade8a35c-kube-api-access-6xq8v\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:27.681517 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.681419    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:27.856795 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:27.856759    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:28.008928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:28.008833    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:28.011601 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:28.011576    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls\") pod \"maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:28.208773 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:28.208734    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:28.600371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:28.600324    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"]
Nov 02 02:11:28.600881 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:11:28.600851    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50d6d24_b770_400d_bf1b_c3ceade8a35c.slice/crio-fff56ec4486a29d82e0a3172f31f377979eb3e5ba92baa71f6e00341874ad602 WatchSource:0}: Error finding container fff56ec4486a29d82e0a3172f31f377979eb3e5ba92baa71f6e00341874ad602: Status 404 returned error can't find the container with id fff56ec4486a29d82e0a3172f31f377979eb3e5ba92baa71f6e00341874ad602
Nov 02 02:11:28.602379 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:28.602362    4912 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:11:29.444611 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:29.444573    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk" event={"ID":"e50d6d24-b770-400d-bf1b-c3ceade8a35c","Type":"ContainerStarted","Data":"fff56ec4486a29d82e0a3172f31f377979eb3e5ba92baa71f6e00341874ad602"}
Nov 02 02:11:30.455923 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:30.455873    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk" event={"ID":"e50d6d24-b770-400d-bf1b-c3ceade8a35c","Type":"ContainerStarted","Data":"229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f"}
Nov 02 02:11:30.456421 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:30.455966    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:30.574547 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:30.574494    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk" podStartSLOduration=2.668245788 podStartE2EDuration="3.57447857s" totalImagesPullingTime="906.232782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:11:30.572719711 +0000 UTC m=+2383.478185071" watchObservedRunningTime="2026-11-02 02:11:30.57447857 +0000 UTC m=+2383.479943927"
Nov 02 02:11:31.202133 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:31.202100    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:31.215808 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:31.215778    4912 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:31.477662 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:31.477585    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:11:32.304945 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.304913    4912 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 02:11:32.305232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.304974    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 02:11:32.510201 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.510161    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:11:32.514736 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.514701    4912 generic.go:356] "Generic (PLEG): container finished" podID="6995d05c-1099-418f-9905-c65248d5c660" containerID="0c02136be1ecd6c5dd2ce10fbe70974b4e41edea483839e8363769071d161881" exitCode=0
Nov 02 02:11:32.515037 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.515009    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" event={"ID":"6995d05c-1099-418f-9905-c65248d5c660","Type":"ContainerDied","Data":"0c02136be1ecd6c5dd2ce10fbe70974b4e41edea483839e8363769071d161881"}
Nov 02 02:11:32.517799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.517767    4912 generic.go:356] "Generic (PLEG): container finished" podID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerID="a0c2701b6ca1bb6293c0385e95ffc654ab995fbb141404197614163db611cf5f" exitCode=0
Nov 02 02:11:32.517921 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.517835    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" event={"ID":"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4","Type":"ContainerDied","Data":"a0c2701b6ca1bb6293c0385e95ffc654ab995fbb141404197614163db611cf5f"}
Nov 02 02:11:32.519807 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.519780    4912 generic.go:356] "Generic (PLEG): container finished" podID="b19ab8da-5870-43b6-be66-998a9c154b93" containerID="e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0" exitCode=0
Nov 02 02:11:32.520651 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.520627    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"
Nov 02 02:11:32.521647 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.521618    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" event={"ID":"b19ab8da-5870-43b6-be66-998a9c154b93","Type":"ContainerDied","Data":"e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0"}
Nov 02 02:11:32.521748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.521696    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5" event={"ID":"b19ab8da-5870-43b6-be66-998a9c154b93","Type":"ContainerDied","Data":"595ef76046092bd56b33abde93e25ceca6e5beb9e1f056de8205c5da1f90a944"}
Nov 02 02:11:32.521815 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.521751    4912 scope.go:122] "RemoveContainer" containerID="e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0"
Nov 02 02:11:32.546430 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.546380    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:32.546587 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.546569    4912 scope.go:122] "RemoveContainer" containerID="e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0"
Nov 02 02:11:32.546911 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:11:32.546884    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0\": container with ID starting with e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0 not found: ID does not exist" containerID="e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0"
Nov 02 02:11:32.547009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.546926    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0"} err="failed to get container status \"e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0\": rpc error: code = NotFound desc = could not find container \"e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0\": container with ID starting with e6fbffa79c8a9f003ceb2b3755dad5d8826d54b848ce016c04647ba52c3f37d0 not found: ID does not exist"
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587029    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-certs\") pod \"b19ab8da-5870-43b6-be66-998a9c154b93\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") "
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587107    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-istio-token\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-istio-token\") pod \"b19ab8da-5870-43b6-be66-998a9c154b93\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") "
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587151    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-envoy\") pod \"b19ab8da-5870-43b6-be66-998a9c154b93\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") "
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587229    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-credential-socket\") pod \"b19ab8da-5870-43b6-be66-998a9c154b93\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") "
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587284    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b19ab8da-5870-43b6-be66-998a9c154b93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19ab8da-5870-43b6-be66-998a9c154b93-istiod-ca-cert\") pod \"b19ab8da-5870-43b6-be66-998a9c154b93\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") "
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587320    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-socket\") pod \"b19ab8da-5870-43b6-be66-998a9c154b93\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") "
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587343    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-certs" pod "b19ab8da-5870-43b6-be66-998a9c154b93" (UID: "b19ab8da-5870-43b6-be66-998a9c154b93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587365    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-kube-api-access-87nb4\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-kube-api-access-87nb4\") pod \"b19ab8da-5870-43b6-be66-998a9c154b93\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") "
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587396    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b19ab8da-5870-43b6-be66-998a9c154b93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19ab8da-5870-43b6-be66-998a9c154b93-istio-podinfo\") pod \"b19ab8da-5870-43b6-be66-998a9c154b93\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") "
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587426    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-data\") pod \"b19ab8da-5870-43b6-be66-998a9c154b93\" (UID: \"b19ab8da-5870-43b6-be66-998a9c154b93\") "
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587661    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19ab8da-5870-43b6-be66-998a9c154b93-istiod-ca-cert" pod "b19ab8da-5870-43b6-be66-998a9c154b93" (UID: "b19ab8da-5870-43b6-be66-998a9c154b93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.587944    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-data" pod "b19ab8da-5870-43b6-be66-998a9c154b93" (UID: "b19ab8da-5870-43b6-be66-998a9c154b93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.588094    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-socket" pod "b19ab8da-5870-43b6-be66-998a9c154b93" (UID: "b19ab8da-5870-43b6-be66-998a9c154b93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.588537    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-credential-socket" pod "b19ab8da-5870-43b6-be66-998a9c154b93" (UID: "b19ab8da-5870-43b6-be66-998a9c154b93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.589332    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.589355    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19ab8da-5870-43b6-be66-998a9c154b93-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.589371    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.589422    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.589436    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.589495    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:11:32.590110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.589747    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-envoy" pod "b19ab8da-5870-43b6-be66-998a9c154b93" (UID: "b19ab8da-5870-43b6-be66-998a9c154b93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.591566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.590162    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-istio-token" pod "b19ab8da-5870-43b6-be66-998a9c154b93" (UID: "b19ab8da-5870-43b6-be66-998a9c154b93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.591566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.590396    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b19ab8da-5870-43b6-be66-998a9c154b93-istio-podinfo" pod "b19ab8da-5870-43b6-be66-998a9c154b93" (UID: "b19ab8da-5870-43b6-be66-998a9c154b93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:11:32.591566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.591270    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-kube-api-access-87nb4" pod "b19ab8da-5870-43b6-be66-998a9c154b93" (UID: "b19ab8da-5870-43b6-be66-998a9c154b93"). InnerVolumeSpecName "kube-api-access-87nb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.592282 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.592260    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:11:32.690245 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690153    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-token\") pod \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") "
Nov 02 02:11:32.690245 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690216    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-certs\") pod \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") "
Nov 02 02:11:32.690489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690246    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-envoy\") pod \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") "
Nov 02 02:11:32.690489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690275    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-socket\") pod \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") "
Nov 02 02:11:32.690489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690303    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-kube-api-access-ms4zs\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-kube-api-access-ms4zs\") pod \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") "
Nov 02 02:11:32.690489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690331    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-data\") pod \"6995d05c-1099-418f-9905-c65248d5c660\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") "
Nov 02 02:11:32.690489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690402    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6995d05c-1099-418f-9905-c65248d5c660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6995d05c-1099-418f-9905-c65248d5c660-istio-podinfo\") pod \"6995d05c-1099-418f-9905-c65248d5c660\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") "
Nov 02 02:11:32.690489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690435    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-envoy\") pod \"6995d05c-1099-418f-9905-c65248d5c660\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") "
Nov 02 02:11:32.690489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690471    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-kube-api-access-pzl5p\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-kube-api-access-pzl5p\") pod \"6995d05c-1099-418f-9905-c65248d5c660\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") "
Nov 02 02:11:32.690862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690523    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-data\") pod \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") "
Nov 02 02:11:32.690862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690552    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-istio-token\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-istio-token\") pod \"6995d05c-1099-418f-9905-c65248d5c660\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") "
Nov 02 02:11:32.690862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690577    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-certs" pod "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" (UID: "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.690862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690593    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-podinfo\") pod \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") "
Nov 02 02:11:32.690862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690649    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-credential-socket\") pod \"6995d05c-1099-418f-9905-c65248d5c660\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") "
Nov 02 02:11:32.690862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690699    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-socket\") pod \"6995d05c-1099-418f-9905-c65248d5c660\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") "
Nov 02 02:11:32.690862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690746    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-credential-socket\") pod \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") "
Nov 02 02:11:32.690862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690822    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-data" pod "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" (UID: "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.690862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690831    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istiod-ca-cert\") pod \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\" (UID: \"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4\") "
Nov 02 02:11:32.691371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690902    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-certs\") pod \"6995d05c-1099-418f-9905-c65248d5c660\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") "
Nov 02 02:11:32.691371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.690966    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6995d05c-1099-418f-9905-c65248d5c660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6995d05c-1099-418f-9905-c65248d5c660-istiod-ca-cert\") pod \"6995d05c-1099-418f-9905-c65248d5c660\" (UID: \"6995d05c-1099-418f-9905-c65248d5c660\") "
Nov 02 02:11:32.691371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.691188    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istiod-ca-cert" pod "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" (UID: "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:32.691722 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.691702    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-credential-socket" pod "6995d05c-1099-418f-9905-c65248d5c660" (UID: "6995d05c-1099-418f-9905-c65248d5c660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.691822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.691755    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6995d05c-1099-418f-9905-c65248d5c660-istiod-ca-cert" pod "6995d05c-1099-418f-9905-c65248d5c660" (UID: "6995d05c-1099-418f-9905-c65248d5c660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:32.691915 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.691897    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-socket" pod "6995d05c-1099-418f-9905-c65248d5c660" (UID: "6995d05c-1099-418f-9905-c65248d5c660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.692113 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692093    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-credential-socket" pod "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" (UID: "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.692363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692341    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692371    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19ab8da-5870-43b6-be66-998a9c154b93-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692386    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692399    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6995d05c-1099-418f-9905-c65248d5c660-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692414    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692428    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87nb4\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-kube-api-access-87nb4\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692443    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19ab8da-5870-43b6-be66-998a9c154b93-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692456    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692469    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692483    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19ab8da-5870-43b6-be66-998a9c154b93-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.692853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.692498    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.693160 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.693111    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-data" pod "6995d05c-1099-418f-9905-c65248d5c660" (UID: "6995d05c-1099-418f-9905-c65248d5c660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.693382 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.693322    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-certs" pod "6995d05c-1099-418f-9905-c65248d5c660" (UID: "6995d05c-1099-418f-9905-c65248d5c660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.693382 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.693348    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-socket" pod "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" (UID: "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.694351 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.694324    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-kube-api-access-ms4zs" pod "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" (UID: "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4"). InnerVolumeSpecName "kube-api-access-ms4zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.694464 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.694440    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6995d05c-1099-418f-9905-c65248d5c660-istio-podinfo" pod "6995d05c-1099-418f-9905-c65248d5c660" (UID: "6995d05c-1099-418f-9905-c65248d5c660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:11:32.694559 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.694538    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-envoy" pod "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" (UID: "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.695240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.695212    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-istio-token" pod "6995d05c-1099-418f-9905-c65248d5c660" (UID: "6995d05c-1099-418f-9905-c65248d5c660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.695690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.695652    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-kube-api-access-pzl5p" pod "6995d05c-1099-418f-9905-c65248d5c660" (UID: "6995d05c-1099-418f-9905-c65248d5c660"). InnerVolumeSpecName "kube-api-access-pzl5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.695864 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.695843    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-podinfo" pod "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" (UID: "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:11:32.695994 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.695975    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-envoy" pod "6995d05c-1099-418f-9905-c65248d5c660" (UID: "6995d05c-1099-418f-9905-c65248d5c660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.696727 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.696697    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-token" pod "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" (UID: "b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.793888 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.793849    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.793888 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.793892    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.794235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.793907    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.794235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.793923    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.794235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.793938    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms4zs\" (UniqueName: \"kubernetes.io/projected/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-kube-api-access-ms4zs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.794235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.793953    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.794235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.793966    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6995d05c-1099-418f-9905-c65248d5c660-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.794235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.793979    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6995d05c-1099-418f-9905-c65248d5c660-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.794235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.793993    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzl5p\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-kube-api-access-pzl5p\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.794235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.794006    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6995d05c-1099-418f-9905-c65248d5c660-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.794235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:32.794020    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:33.088542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.088451    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"]
Nov 02 02:11:33.094302 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.094272    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-dk9f5"]
Nov 02 02:11:33.227907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.227875    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" path="/var/lib/kubelet/pods/b19ab8da-5870-43b6-be66-998a9c154b93/volumes"
Nov 02 02:11:33.529856 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.529815    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"
Nov 02 02:11:33.530345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.529817    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj" event={"ID":"6995d05c-1099-418f-9905-c65248d5c660","Type":"ContainerDied","Data":"b84de789a812f020736ab8f5d8f3a0e75e146f996972b23831edfdd79d622d3c"}
Nov 02 02:11:33.530345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.529992    4912 scope.go:122] "RemoveContainer" containerID="0c02136be1ecd6c5dd2ce10fbe70974b4e41edea483839e8363769071d161881"
Nov 02 02:11:33.531572 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.531560    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"
Nov 02 02:11:33.531680 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.531560    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl" event={"ID":"b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4","Type":"ContainerDied","Data":"515a6cb4d8cc02202c836513b255a5f23f2aaab7c7bb19f1c0d45594d58e38f2"}
Nov 02 02:11:33.544118 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.544099    4912 scope.go:122] "RemoveContainer" containerID="a0c2701b6ca1bb6293c0385e95ffc654ab995fbb141404197614163db611cf5f"
Nov 02 02:11:33.696872 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.696840    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"]
Nov 02 02:11:33.746659 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.746629    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-rlkbj"]
Nov 02 02:11:33.808682 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.808601    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"]
Nov 02 02:11:33.812422 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:33.812398    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-v62kl"]
Nov 02 02:11:34.570484 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:34.570451    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:34.795706 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:34.795671    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:35.227036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:35.227004    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6995d05c-1099-418f-9905-c65248d5c660" path="/var/lib/kubelet/pods/6995d05c-1099-418f-9905-c65248d5c660/volumes"
Nov 02 02:11:35.227523 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:35.227506    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" path="/var/lib/kubelet/pods/b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4/volumes"
Nov 02 02:11:36.471117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:36.471084    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:11:37.363635 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:37.363587    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:42.138204 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:11:42.138166    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:11:42.158546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:42.158511    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:46.938598 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:46.938554    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:47.786458 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:47.786430    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:11:47.793039 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:47.793017    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:11:47.916935 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:47.916907    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:11:47.923571 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:47.923552    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:11:48.961480 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:48.961446    4912 scope.go:122] "RemoveContainer" containerID="43a727922f5c5c6ddb83b15c32d8d65f200f405b01e670f4a37bc1df88fd5360"
Nov 02 02:11:48.975003 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:48.974975    4912 scope.go:122] "RemoveContainer" containerID="9cec03395b9baad35a753223b71bdf7641b2f6e9b452bed8c8366de817b69c5e"
Nov 02 02:11:48.986104 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:48.986079    4912 scope.go:122] "RemoveContainer" containerID="45b51ebd55fbf66e988ced770ca8374b6723fd39f2cd4ef3382ac9aa6192a314"
Nov 02 02:11:51.583648 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:51.583605    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:51.757159 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:51.757118    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:56.073082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:56.072211    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"]
Nov 02 02:11:56.110083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:56.109665    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b5d4d-cbm25"]
Nov 02 02:11:56.819081 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:56.818050    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:11:56.858590 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:56.858541    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-fhsck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:11:56.858919 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:56.858885    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podUID="da8141a6-dfcd-4672-a751-1d2f60d39bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:11:57.226668 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:11:57.226633    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c922f73-2e92-4ea2-895b-f0b9ac10add4" path="/var/lib/kubelet/pods/6c922f73-2e92-4ea2-895b-f0b9ac10add4/volumes"
Nov 02 02:12:01.773081 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:01.772459    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:06.535207 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:06.535169    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:11.460970 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:11.460931    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:12.140339 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:12:12.140302    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:12:16.232325 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:16.232284    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:19.940767 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:19.940728    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:21.216478 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:21.216432    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:24.770300 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:24.770256    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:26.341875 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:26.341838    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:29.851705 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:29.851656    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:31.341520 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:31.341484    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:32.985884 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:32.985837    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:33.162270 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:33.162227    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:36.148814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:36.148782    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:36.375701 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:36.375653    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:37.950521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:37.950486    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"]
Nov 02 02:12:38.228937 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:38.228854    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:38.415383 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:38.415327    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:41.049579 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:41.049536    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:41.159387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:41.159351    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"]
Nov 02 02:12:41.208310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:41.208275    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"]
Nov 02 02:12:41.212079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:41.212028    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"]
Nov 02 02:12:41.293174 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:41.293134    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:41.474162 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:41.474124    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:42.138795 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:12:42.138754    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:12:42.951629 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:42.951590    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7" podUID="039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" containerName="payload-pre-processing" containerID="cri-o://7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223" gracePeriod=25
Nov 02 02:12:46.254741 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:46.254695    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:48.174535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:48.174500    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"]
Nov 02 02:12:48.565832 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:48.565739    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:48.736294 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:48.736257    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:49.043962 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:49.043927    4912 scope.go:122] "RemoveContainer" containerID="5599ff2565ff8c860ecaa20da7aaab84969470bd5559bf64a224c8d14dea2192"
Nov 02 02:12:49.057770 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:49.057748    4912 scope.go:122] "RemoveContainer" containerID="c5a9218ae4d1c4b44b78d82c5522892bedd0af2ee09dd3c1a018b7e9adf88f52"
Nov 02 02:12:49.068848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:49.068821    4912 scope.go:122] "RemoveContainer" containerID="8b4236c21a8e2ceb479421f260ee58da5f8cb4a7fcaa492229ec7876c2ed2571"
Nov 02 02:12:50.673323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:50.673276    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7" podUID="039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:12:51.062889 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:51.062788    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:51.928687 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:51.928632    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:53.175202 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:53.175129    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69" podUID="59b93040-c359-4e41-a8ff-923846973e4a" containerName="payload-processing" containerID="cri-o://d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740" gracePeriod=25
Nov 02 02:12:55.934504 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:55.934466    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69" podUID="59b93040-c359-4e41-a8ff-923846973e4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:12:56.029284 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.029248    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:12:56.230689 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.230600    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main" containerID="cri-o://efccaa57cc844c3f85de106c91efa468a1aae678e088b1301705462ba7458669" gracePeriod=45
Nov 02 02:12:56.234198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.234149    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main" containerID="cri-o://67c2ec2b8ecc3b68dfe80d4d269477797b26089eda70923ca1dda36b04bba890" gracePeriod=45
Nov 02 02:12:56.252943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.252909    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main" containerID="cri-o://3688e2dfb5919339d31406bf7837cd00604f4e2d0da2e89b0c70b93ff0390164" gracePeriod=45
Nov 02 02:12:56.424583 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.424461    4912 generic.go:356] "Generic (PLEG): container finished" podID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerID="efccaa57cc844c3f85de106c91efa468a1aae678e088b1301705462ba7458669" exitCode=0
Nov 02 02:12:56.424583 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.424529    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" event={"ID":"f6a49295-57be-4bb4-9148-5bde151f6ecb","Type":"ContainerDied","Data":"efccaa57cc844c3f85de106c91efa468a1aae678e088b1301705462ba7458669"}
Nov 02 02:12:56.426232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.426122    4912 generic.go:356] "Generic (PLEG): container finished" podID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerID="3688e2dfb5919339d31406bf7837cd00604f4e2d0da2e89b0c70b93ff0390164" exitCode=0
Nov 02 02:12:56.426232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.426190    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" event={"ID":"d3c07e88-8d03-43bb-b28f-e3ef7490775b","Type":"ContainerDied","Data":"3688e2dfb5919339d31406bf7837cd00604f4e2d0da2e89b0c70b93ff0390164"}
Nov 02 02:12:56.429177 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.428127    4912 generic.go:356] "Generic (PLEG): container finished" podID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerID="67c2ec2b8ecc3b68dfe80d4d269477797b26089eda70923ca1dda36b04bba890" exitCode=0
Nov 02 02:12:56.429177 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.428177    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" event={"ID":"569172aa-26cb-46aa-8a18-d46f3c7630c6","Type":"ContainerDied","Data":"67c2ec2b8ecc3b68dfe80d4d269477797b26089eda70923ca1dda36b04bba890"}
Nov 02 02:12:56.550182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.550150    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 02:12:56.574615 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.574588    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 02:12:56.578735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.578706    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 02:12:56.661125 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661089    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a49295-57be-4bb4-9148-5bde151f6ecb-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a49295-57be-4bb4-9148-5bde151f6ecb-tls-certs\") pod \"f6a49295-57be-4bb4-9148-5bde151f6ecb\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") "
Nov 02 02:12:56.661322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661147    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-dshm\") pod \"f6a49295-57be-4bb4-9148-5bde151f6ecb\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") "
Nov 02 02:12:56.661322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661180    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-tmp-dir\") pod \"f6a49295-57be-4bb4-9148-5bde151f6ecb\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") "
Nov 02 02:12:56.661322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661213    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-model-cache\") pod \"f6a49295-57be-4bb4-9148-5bde151f6ecb\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") "
Nov 02 02:12:56.661322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661243    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-model-cache\") pod \"569172aa-26cb-46aa-8a18-d46f3c7630c6\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") "
Nov 02 02:12:56.661322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661288    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a49295-57be-4bb4-9148-5bde151f6ecb-kube-api-access-mzqhr\" (UniqueName: \"kubernetes.io/projected/f6a49295-57be-4bb4-9148-5bde151f6ecb-kube-api-access-mzqhr\") pod \"f6a49295-57be-4bb4-9148-5bde151f6ecb\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") "
Nov 02 02:12:56.661608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661353    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569172aa-26cb-46aa-8a18-d46f3c7630c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/569172aa-26cb-46aa-8a18-d46f3c7630c6-tls-certs\") pod \"569172aa-26cb-46aa-8a18-d46f3c7630c6\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") "
Nov 02 02:12:56.661608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661416    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569172aa-26cb-46aa-8a18-d46f3c7630c6-kube-api-access-5rkf6\" (UniqueName: \"kubernetes.io/projected/569172aa-26cb-46aa-8a18-d46f3c7630c6-kube-api-access-5rkf6\") pod \"569172aa-26cb-46aa-8a18-d46f3c7630c6\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") "
Nov 02 02:12:56.661608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661467    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-home\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-home\") pod \"f6a49295-57be-4bb4-9148-5bde151f6ecb\" (UID: \"f6a49295-57be-4bb4-9148-5bde151f6ecb\") "
Nov 02 02:12:56.661608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661539    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-tmp-dir\") pod \"569172aa-26cb-46aa-8a18-d46f3c7630c6\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") "
Nov 02 02:12:56.661608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661542    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-tmp-dir" pod "f6a49295-57be-4bb4-9148-5bde151f6ecb" (UID: "f6a49295-57be-4bb4-9148-5bde151f6ecb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.661899 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661617    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-dshm\") pod \"569172aa-26cb-46aa-8a18-d46f3c7630c6\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") "
Nov 02 02:12:56.661899 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661651    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-home\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-home\") pod \"569172aa-26cb-46aa-8a18-d46f3c7630c6\" (UID: \"569172aa-26cb-46aa-8a18-d46f3c7630c6\") "
Nov 02 02:12:56.661899 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661756    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-model-cache" pod "f6a49295-57be-4bb4-9148-5bde151f6ecb" (UID: "f6a49295-57be-4bb4-9148-5bde151f6ecb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.662088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.661978    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-home" pod "f6a49295-57be-4bb4-9148-5bde151f6ecb" (UID: "f6a49295-57be-4bb4-9148-5bde151f6ecb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.662535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.662383    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.662535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.662412    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.662535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.662430    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-tmp-dir" pod "569172aa-26cb-46aa-8a18-d46f3c7630c6" (UID: "569172aa-26cb-46aa-8a18-d46f3c7630c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.662535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.662453    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.662535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.662502    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-model-cache" pod "569172aa-26cb-46aa-8a18-d46f3c7630c6" (UID: "569172aa-26cb-46aa-8a18-d46f3c7630c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.663205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.662678    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-home" pod "569172aa-26cb-46aa-8a18-d46f3c7630c6" (UID: "569172aa-26cb-46aa-8a18-d46f3c7630c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.663953 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.663921    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a49295-57be-4bb4-9148-5bde151f6ecb-tls-certs" pod "f6a49295-57be-4bb4-9148-5bde151f6ecb" (UID: "f6a49295-57be-4bb4-9148-5bde151f6ecb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:56.664159 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.664138    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-dshm" pod "f6a49295-57be-4bb4-9148-5bde151f6ecb" (UID: "f6a49295-57be-4bb4-9148-5bde151f6ecb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.664577 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.664554    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a49295-57be-4bb4-9148-5bde151f6ecb-kube-api-access-mzqhr" pod "f6a49295-57be-4bb4-9148-5bde151f6ecb" (UID: "f6a49295-57be-4bb4-9148-5bde151f6ecb"). InnerVolumeSpecName "kube-api-access-mzqhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:56.665141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.665104    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569172aa-26cb-46aa-8a18-d46f3c7630c6-kube-api-access-5rkf6" pod "569172aa-26cb-46aa-8a18-d46f3c7630c6" (UID: "569172aa-26cb-46aa-8a18-d46f3c7630c6"). InnerVolumeSpecName "kube-api-access-5rkf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:56.665442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.665408    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-dshm" pod "569172aa-26cb-46aa-8a18-d46f3c7630c6" (UID: "569172aa-26cb-46aa-8a18-d46f3c7630c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.671170 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.671138    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569172aa-26cb-46aa-8a18-d46f3c7630c6-tls-certs" pod "569172aa-26cb-46aa-8a18-d46f3c7630c6" (UID: "569172aa-26cb-46aa-8a18-d46f3c7630c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:56.763246 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.763207    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-dshm\") pod \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") "
Nov 02 02:12:56.763477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.763279    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tls-certs\") pod \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") "
Nov 02 02:12:56.763477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.763329    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-model-cache\") pod \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") "
Nov 02 02:12:56.763477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.763417    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c07e88-8d03-43bb-b28f-e3ef7490775b-kube-api-access-hwvnr\" (UniqueName: \"kubernetes.io/projected/d3c07e88-8d03-43bb-b28f-e3ef7490775b-kube-api-access-hwvnr\") pod \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") "
Nov 02 02:12:56.763743 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.763510    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tmp-dir\") pod \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") "
Nov 02 02:12:56.763743 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.763564    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-home\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-home\") pod \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\" (UID: \"d3c07e88-8d03-43bb-b28f-e3ef7490775b\") "
Nov 02 02:12:56.763743 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.763660    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-model-cache" pod "d3c07e88-8d03-43bb-b28f-e3ef7490775b" (UID: "d3c07e88-8d03-43bb-b28f-e3ef7490775b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.763977 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.763825    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tmp-dir" pod "d3c07e88-8d03-43bb-b28f-e3ef7490775b" (UID: "d3c07e88-8d03-43bb-b28f-e3ef7490775b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.764050 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764004    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-home" pod "d3c07e88-8d03-43bb-b28f-e3ef7490775b" (UID: "d3c07e88-8d03-43bb-b28f-e3ef7490775b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.764236 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764158    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764236 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764188    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rkf6\" (UniqueName: \"kubernetes.io/projected/569172aa-26cb-46aa-8a18-d46f3c7630c6-kube-api-access-5rkf6\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764236 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764207    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764236 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764224    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764248    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764261    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764275    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a49295-57be-4bb4-9148-5bde151f6ecb-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764288    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764313    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a49295-57be-4bb4-9148-5bde151f6ecb-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764347    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569172aa-26cb-46aa-8a18-d46f3c7630c6-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764363    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzqhr\" (UniqueName: \"kubernetes.io/projected/f6a49295-57be-4bb4-9148-5bde151f6ecb-kube-api-access-mzqhr\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.764594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.764379    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569172aa-26cb-46aa-8a18-d46f3c7630c6-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.765901 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.765876    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tls-certs" pod "d3c07e88-8d03-43bb-b28f-e3ef7490775b" (UID: "d3c07e88-8d03-43bb-b28f-e3ef7490775b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:56.766040 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.765946    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c07e88-8d03-43bb-b28f-e3ef7490775b-kube-api-access-hwvnr" pod "d3c07e88-8d03-43bb-b28f-e3ef7490775b" (UID: "d3c07e88-8d03-43bb-b28f-e3ef7490775b"). InnerVolumeSpecName "kube-api-access-hwvnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:56.766196 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.766179    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-dshm" pod "d3c07e88-8d03-43bb-b28f-e3ef7490775b" (UID: "d3c07e88-8d03-43bb-b28f-e3ef7490775b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:56.865294 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.865267    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c07e88-8d03-43bb-b28f-e3ef7490775b-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.865446 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.865302    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c07e88-8d03-43bb-b28f-e3ef7490775b-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:56.865446 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:56.865320    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvnr\" (UniqueName: \"kubernetes.io/projected/d3c07e88-8d03-43bb-b28f-e3ef7490775b-kube-api-access-hwvnr\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:57.437315 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.437285    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"
Nov 02 02:12:57.437315 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.437300    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj" event={"ID":"f6a49295-57be-4bb4-9148-5bde151f6ecb","Type":"ContainerDied","Data":"1621db7aed7db5519caacb92474eb523a39fe17c97e257e4d5659df0568d2db5"}
Nov 02 02:12:57.437871 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.437353    4912 scope.go:122] "RemoveContainer" containerID="efccaa57cc844c3f85de106c91efa468a1aae678e088b1301705462ba7458669"
Nov 02 02:12:57.439253 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.439218    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"
Nov 02 02:12:57.439416 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.439255    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l" event={"ID":"d3c07e88-8d03-43bb-b28f-e3ef7490775b","Type":"ContainerDied","Data":"e930d258cc1c42635c61e338e718a4c06803ad93a0d53c1fe66790cda5caa96a"}
Nov 02 02:12:57.441613 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.441292    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2" event={"ID":"569172aa-26cb-46aa-8a18-d46f3c7630c6","Type":"ContainerDied","Data":"cb4c002a16e458317b2afd7056f03091c2abcd55fc6728bf58b2f87753737f0b"}
Nov 02 02:12:57.441613 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.441385    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"
Nov 02 02:12:57.451151 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.451123    4912 scope.go:122] "RemoveContainer" containerID="3688e2dfb5919339d31406bf7837cd00604f4e2d0da2e89b0c70b93ff0390164"
Nov 02 02:12:57.462971 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.462949    4912 scope.go:122] "RemoveContainer" containerID="67c2ec2b8ecc3b68dfe80d4d269477797b26089eda70923ca1dda36b04bba890"
Nov 02 02:12:57.602408 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.602312    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"]
Nov 02 02:12:57.606278 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.606251    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-w1-dbe849-kserve-56d4889976-tkhfj"]
Nov 02 02:12:57.715145 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.715038    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"]
Nov 02 02:12:57.719070 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.719023    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/premium-w1-dbe849-kserve-76c4b7cf6f-j4br2"]
Nov 02 02:12:57.828202 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.828009    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"]
Nov 02 02:12:57.832009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:57.831983    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dbe849/distinct-2-w1-dbe849-kserve-c9cfc99d7-kg58l"]
Nov 02 02:12:59.225559 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:59.225528    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" path="/var/lib/kubelet/pods/569172aa-26cb-46aa-8a18-d46f3c7630c6/volumes"
Nov 02 02:12:59.225953 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:59.225933    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" path="/var/lib/kubelet/pods/d3c07e88-8d03-43bb-b28f-e3ef7490775b/volumes"
Nov 02 02:12:59.226340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:12:59.226325    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" path="/var/lib/kubelet/pods/f6a49295-57be-4bb4-9148-5bde151f6ecb/volumes"
Nov 02 02:13:00.673427 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:00.673384    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7" podUID="039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:13:00.845390 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:00.845352    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:03.231668 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.231643    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 02:13:03.241021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.240982    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-plugins-config-volume\") pod \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\" (UID: \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\") "
Nov 02 02:13:03.241204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.241091    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-kube-api-access-pbzkm\" (UniqueName: \"kubernetes.io/projected/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-kube-api-access-pbzkm\") pod \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\" (UID: \"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2\") "
Nov 02 02:13:03.241443 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.241411    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-plugins-config-volume" pod "039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" (UID: "039a37fd-6287-4fb0-b587-dc6fe5bcd7f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:03.241706 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.241666    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:03.243750 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.243727    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-kube-api-access-pbzkm" pod "039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" (UID: "039a37fd-6287-4fb0-b587-dc6fe5bcd7f2"). InnerVolumeSpecName "kube-api-access-pbzkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:03.342850 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.342816    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbzkm\" (UniqueName: \"kubernetes.io/projected/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2-kube-api-access-pbzkm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:03.498098 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.497966    4912 generic.go:356] "Generic (PLEG): container finished" podID="039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" containerID="7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223" exitCode=0
Nov 02 02:13:03.498098 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.498014    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"
Nov 02 02:13:03.498313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.498268    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7" event={"ID":"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2","Type":"ContainerDied","Data":"7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223"}
Nov 02 02:13:03.498313 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.498307    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7" event={"ID":"039a37fd-6287-4fb0-b587-dc6fe5bcd7f2","Type":"ContainerDied","Data":"56cbd9e5ebd98420f6d8fa1f966acb48bbd432f6148478e10a753e32127b2af7"}
Nov 02 02:13:03.498407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.498328    4912 scope.go:122] "RemoveContainer" containerID="7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223"
Nov 02 02:13:03.509916 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.509899    4912 scope.go:122] "RemoveContainer" containerID="7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223"
Nov 02 02:13:03.510260 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:13:03.510232    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223\": container with ID starting with 7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223 not found: ID does not exist" containerID="7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223"
Nov 02 02:13:03.510347 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.510273    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223"} err="failed to get container status \"7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223\": rpc error: code = NotFound desc = could not find container \"7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223\": container with ID starting with 7867cf9461739dca0933954e20a679a44b09d4b62d04ea00dde2a919a92d4223 not found: ID does not exist"
Nov 02 02:13:03.663302 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.663265    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"]
Nov 02 02:13:03.668226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:03.668196    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9470a3-c4476697f-h8dz7"]
Nov 02 02:13:05.225372 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:05.225332    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" path="/var/lib/kubelet/pods/039a37fd-6287-4fb0-b587-dc6fe5bcd7f2/volumes"
Nov 02 02:13:05.933667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:05.933632    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69" podUID="59b93040-c359-4e41-a8ff-923846973e4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:13:05.993399 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:05.993359    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:06.165895 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:06.165859    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:07.989797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:07.989749    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:10.651844 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:10.651791    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:12.137907 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:13:12.137868    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:13:13.350581 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.350554    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 02:13:13.463400 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.463315    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b93040-c359-4e41-a8ff-923846973e4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b93040-c359-4e41-a8ff-923846973e4a-plugins-config-volume\") pod \"59b93040-c359-4e41-a8ff-923846973e4a\" (UID: \"59b93040-c359-4e41-a8ff-923846973e4a\") "
Nov 02 02:13:13.463564 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.463419    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b93040-c359-4e41-a8ff-923846973e4a-kube-api-access-9tgc4\" (UniqueName: \"kubernetes.io/projected/59b93040-c359-4e41-a8ff-923846973e4a-kube-api-access-9tgc4\") pod \"59b93040-c359-4e41-a8ff-923846973e4a\" (UID: \"59b93040-c359-4e41-a8ff-923846973e4a\") "
Nov 02 02:13:13.463720 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.463697    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b93040-c359-4e41-a8ff-923846973e4a-plugins-config-volume" pod "59b93040-c359-4e41-a8ff-923846973e4a" (UID: "59b93040-c359-4e41-a8ff-923846973e4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:13.463842 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.463826    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b93040-c359-4e41-a8ff-923846973e4a-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:13.466498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.466472    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b93040-c359-4e41-a8ff-923846973e4a-kube-api-access-9tgc4" pod "59b93040-c359-4e41-a8ff-923846973e4a" (UID: "59b93040-c359-4e41-a8ff-923846973e4a"). InnerVolumeSpecName "kube-api-access-9tgc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:13.564778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.564749    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tgc4\" (UniqueName: \"kubernetes.io/projected/59b93040-c359-4e41-a8ff-923846973e4a-kube-api-access-9tgc4\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:13.579531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.579490    4912 generic.go:356] "Generic (PLEG): container finished" podID="59b93040-c359-4e41-a8ff-923846973e4a" containerID="d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740" exitCode=0
Nov 02 02:13:13.579706 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.579565    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"
Nov 02 02:13:13.579706 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.579666    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69" event={"ID":"59b93040-c359-4e41-a8ff-923846973e4a","Type":"ContainerDied","Data":"d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740"}
Nov 02 02:13:13.579850 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.579715    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69" event={"ID":"59b93040-c359-4e41-a8ff-923846973e4a","Type":"ContainerDied","Data":"d27d0adf050a2c992e3d22bbd1e9dd4f5e3ceed2f932f3ab2be3595472934e38"}
Nov 02 02:13:13.579850 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.579749    4912 scope.go:122] "RemoveContainer" containerID="d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740"
Nov 02 02:13:13.592620 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.592574    4912 scope.go:122] "RemoveContainer" containerID="d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740"
Nov 02 02:13:13.593358 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:13:13.593330    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740\": container with ID starting with d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740 not found: ID does not exist" containerID="d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740"
Nov 02 02:13:13.593476 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.593368    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740"} err="failed to get container status \"d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740\": rpc error: code = NotFound desc = could not find container \"d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740\": container with ID starting with d8e47c915104e6442c4fd5aacbc8eab94226b29bc261fc193f8bcf67fe6d0740 not found: ID does not exist"
Nov 02 02:13:13.745159 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.745076    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"]
Nov 02 02:13:13.750725 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:13.750690    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dbe849-78dcfdf457-dhl69"]
Nov 02 02:13:14.002862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:14.002772    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:14.211972 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:14.211943    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:14.906495 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:14.906465    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"]
Nov 02 02:13:14.906960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:14.906674    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh" podUID="d1ce6436-0b1f-4f8d-8e52-112531e29105" containerName="maas-api" containerID="cri-o://daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10" gracePeriod=30
Nov 02 02:13:15.035492 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.035458    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:15.208926 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.208901    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:13:15.225995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.225964    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b93040-c359-4e41-a8ff-923846973e4a" path="/var/lib/kubelet/pods/59b93040-c359-4e41-a8ff-923846973e4a/volumes"
Nov 02 02:13:15.297082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.290726    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ce6436-0b1f-4f8d-8e52-112531e29105-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce6436-0b1f-4f8d-8e52-112531e29105-openshift-service-ca\") pod \"d1ce6436-0b1f-4f8d-8e52-112531e29105\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") "
Nov 02 02:13:15.297082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.290834    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-metrics-certs\") pod \"d1ce6436-0b1f-4f8d-8e52-112531e29105\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") "
Nov 02 02:13:15.297082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.291005    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ce6436-0b1f-4f8d-8e52-112531e29105-kube-api-access-kljg6\" (UniqueName: \"kubernetes.io/projected/d1ce6436-0b1f-4f8d-8e52-112531e29105-kube-api-access-kljg6\") pod \"d1ce6436-0b1f-4f8d-8e52-112531e29105\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") "
Nov 02 02:13:15.297082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.291076    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls\") pod \"d1ce6436-0b1f-4f8d-8e52-112531e29105\" (UID: \"d1ce6436-0b1f-4f8d-8e52-112531e29105\") "
Nov 02 02:13:15.297082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.293568    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ce6436-0b1f-4f8d-8e52-112531e29105-openshift-service-ca" pod "d1ce6436-0b1f-4f8d-8e52-112531e29105" (UID: "d1ce6436-0b1f-4f8d-8e52-112531e29105"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:15.310205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.300511    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ce6436-0b1f-4f8d-8e52-112531e29105-kube-api-access-kljg6" pod "d1ce6436-0b1f-4f8d-8e52-112531e29105" (UID: "d1ce6436-0b1f-4f8d-8e52-112531e29105"). InnerVolumeSpecName "kube-api-access-kljg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:15.310205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.309781    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls" pod "d1ce6436-0b1f-4f8d-8e52-112531e29105" (UID: "d1ce6436-0b1f-4f8d-8e52-112531e29105"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:13:15.310594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.310245    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-metrics-certs" pod "d1ce6436-0b1f-4f8d-8e52-112531e29105" (UID: "d1ce6436-0b1f-4f8d-8e52-112531e29105"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:13:15.392254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.392222    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kljg6\" (UniqueName: \"kubernetes.io/projected/d1ce6436-0b1f-4f8d-8e52-112531e29105-kube-api-access-kljg6\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:15.392254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.392254    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:15.392475 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.392265    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce6436-0b1f-4f8d-8e52-112531e29105-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:15.392475 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.392274    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ce6436-0b1f-4f8d-8e52-112531e29105-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:15.599285 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.599195    4912 generic.go:356] "Generic (PLEG): container finished" podID="d1ce6436-0b1f-4f8d-8e52-112531e29105" containerID="daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10" exitCode=0
Nov 02 02:13:15.599285 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.599240    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"
Nov 02 02:13:15.599553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.599287    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh" event={"ID":"d1ce6436-0b1f-4f8d-8e52-112531e29105","Type":"ContainerDied","Data":"daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10"}
Nov 02 02:13:15.599553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.599347    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh" event={"ID":"d1ce6436-0b1f-4f8d-8e52-112531e29105","Type":"ContainerDied","Data":"e3d3088e1c615466e13143364926e3256ff96acca47dcad6bbad4fe40e2a848c"}
Nov 02 02:13:15.599553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.599375    4912 scope.go:122] "RemoveContainer" containerID="daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10"
Nov 02 02:13:15.610680 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.610664    4912 scope.go:122] "RemoveContainer" containerID="daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10"
Nov 02 02:13:15.610976 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:13:15.610959    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10\": container with ID starting with daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10 not found: ID does not exist" containerID="daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10"
Nov 02 02:13:15.611027 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.610987    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10"} err="failed to get container status \"daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10\": rpc error: code = NotFound desc = could not find container \"daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10\": container with ID starting with daf9d2ba7649cf7f2d3402b7976de0b7049baa2e3a6eff9f5283e90aafae0b10 not found: ID does not exist"
Nov 02 02:13:15.764778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.764736    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"]
Nov 02 02:13:15.768206 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:15.768178    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092079-6f99f4c8bd-78rlh"]
Nov 02 02:13:16.537542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:16.537506    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:16.753691 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:16.753653    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:16.929910 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:16.929879    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:17.227411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:17.227324    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ce6436-0b1f-4f8d-8e52-112531e29105" path="/var/lib/kubelet/pods/d1ce6436-0b1f-4f8d-8e52-112531e29105/volumes"
Nov 02 02:13:17.614992 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:17.614949    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:19.416884 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:19.416850    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:19.631887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:19.631851    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:19.866908 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:19.866870    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:21.942775 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:21.942744    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:24.119378 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:24.119343    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:24.345396 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:24.345363    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:24.545192 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:24.545030    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:25.997689 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:25.997644    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:26.990006 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:26.989963    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:31.051605 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:31.051559    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"]
Nov 02 02:13:31.101110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:31.101073    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4326c5bf-vwbbp"]
Nov 02 02:13:31.225524 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:31.225486    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1de6f5e-bbdc-42cb-baf1-ce7f7d795860" path="/var/lib/kubelet/pods/c1de6f5e-bbdc-42cb-baf1-ce7f7d795860/volumes"
Nov 02 02:13:32.049498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:32.049459    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"]
Nov 02 02:13:32.099794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:32.099756    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4417ac7-mjxwh"]
Nov 02 02:13:32.141502 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:32.141455    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:32.326720 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:32.326629    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:33.227845 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:33.227809    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67d002a-e8d3-44a7-b91c-802ceb2f68f9" path="/var/lib/kubelet/pods/f67d002a-e8d3-44a7-b91c-802ceb2f68f9/volumes"
Nov 02 02:13:37.258699 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:37.258659    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:37.483767 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:37.483725    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:41.792496 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:41.792462    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:42.137972 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:13:42.137936    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:13:43.320814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:43.320780    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:43.582755 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:43.582657    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:43.780987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:43.780941    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:44.250802 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:44.250768    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:48.061417 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:48.061378    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:49.117923 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:49.117886    4912 scope.go:122] "RemoveContainer" containerID="ffe5e21773bde94ba3cf3567f0298ff6e58110430ecfd7e95fd57f9ac57df425"
Nov 02 02:13:49.130678 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:49.130652    4912 scope.go:122] "RemoveContainer" containerID="f47bcba3686e96789e679662c630bc0781debaa53410f83dd48b6d3ec28e6e74"
Nov 02 02:13:49.141147 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:49.141126    4912 scope.go:122] "RemoveContainer" containerID="cc4537bd75304114625ac880c2bfa08ee3280f7d72f8e66132cbb22e9b698795"
Nov 02 02:13:49.151720 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:49.151698    4912 scope.go:122] "RemoveContainer" containerID="318a03386fce85b201edd11217acac96a5076cb56836ac66e99c145eca416284"
Nov 02 02:13:49.165069 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:49.165031    4912 scope.go:122] "RemoveContainer" containerID="7d5e3f9de6c1988ceccb243c165e65cfa64097976e5d5821a9dbccf603fc4731"
Nov 02 02:13:49.177510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:49.177482    4912 scope.go:122] "RemoveContainer" containerID="a8cbf4cff3fa13d6d0b0828405de51abfa9338780529fe03e033b69806134614"
Nov 02 02:13:49.189049 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:49.189028    4912 scope.go:122] "RemoveContainer" containerID="20bf14cef2e3e2ae7a9b937cdf61bb244d1bb9deb1c7ac4a108fd69057914f22"
Nov 02 02:13:49.638219 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:49.638173    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:49.866927 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:49.866886    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:50.070155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:50.066987    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:50.569181 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:50.569141    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:50.999237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:50.999200    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:51.179634 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:51.179596    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:51.414916 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:51.414885    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:51.703904 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:51.703810    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:51.740563 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:51.740523    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"]
Nov 02 02:13:51.740856 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:51.740827    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy" containerID="cri-o://56db471be5f5433a711062d6a5e648d4600cf0712a348acc088da87a21fb391e" gracePeriod=30
Nov 02 02:13:51.884309 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:51.884271    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:52.289770 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:52.289732    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:54.940585 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:54.940541    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:56.569690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:56.569652    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:56.754357 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:56.754317    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:56.933223 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:56.932917    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:13:57.025428 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.025395    4912 generic.go:356] "Generic (PLEG): container finished" podID="50954522-828f-43be-9e87-b281ad092a10" containerID="56db471be5f5433a711062d6a5e648d4600cf0712a348acc088da87a21fb391e" exitCode=0
Nov 02 02:13:57.025645 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.025627    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" event={"ID":"50954522-828f-43be-9e87-b281ad092a10","Type":"ContainerDied","Data":"56db471be5f5433a711062d6a5e648d4600cf0712a348acc088da87a21fb391e"}
Nov 02 02:13:57.025750 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.025690    4912 scope.go:122] "RemoveContainer" containerID="f46025c8d4701f51f1946451349886f4ac831953cf4c6a02dc4ad444c1ce0b75"
Nov 02 02:13:57.076918 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.076884    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 02:13:57.135572 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.135539    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-data\") pod \"50954522-828f-43be-9e87-b281ad092a10\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") "
Nov 02 02:13:57.135757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.135592    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-kube-api-access-l48sq\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-kube-api-access-l48sq\") pod \"50954522-828f-43be-9e87-b281ad092a10\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") "
Nov 02 02:13:57.135757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.135656    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-credential-socket\") pod \"50954522-828f-43be-9e87-b281ad092a10\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") "
Nov 02 02:13:57.135757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.135700    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50954522-828f-43be-9e87-b281ad092a10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50954522-828f-43be-9e87-b281ad092a10-istio-podinfo\") pod \"50954522-828f-43be-9e87-b281ad092a10\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") "
Nov 02 02:13:57.135757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.135735    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-istio-token\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-istio-token\") pod \"50954522-828f-43be-9e87-b281ad092a10\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") "
Nov 02 02:13:57.135925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.135847    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50954522-828f-43be-9e87-b281ad092a10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50954522-828f-43be-9e87-b281ad092a10-istiod-ca-cert\") pod \"50954522-828f-43be-9e87-b281ad092a10\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") "
Nov 02 02:13:57.135925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.135877    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-data" pod "50954522-828f-43be-9e87-b281ad092a10" (UID: "50954522-828f-43be-9e87-b281ad092a10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:57.135925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.135891    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-certs\") pod \"50954522-828f-43be-9e87-b281ad092a10\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") "
Nov 02 02:13:57.136124 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136033    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-envoy\") pod \"50954522-828f-43be-9e87-b281ad092a10\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") "
Nov 02 02:13:57.136124 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136037    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-credential-socket" pod "50954522-828f-43be-9e87-b281ad092a10" (UID: "50954522-828f-43be-9e87-b281ad092a10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:57.136249 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136148    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-socket\") pod \"50954522-828f-43be-9e87-b281ad092a10\" (UID: \"50954522-828f-43be-9e87-b281ad092a10\") "
Nov 02 02:13:57.136390 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136148    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-certs" pod "50954522-828f-43be-9e87-b281ad092a10" (UID: "50954522-828f-43be-9e87-b281ad092a10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:57.136390 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136378    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-socket" pod "50954522-828f-43be-9e87-b281ad092a10" (UID: "50954522-828f-43be-9e87-b281ad092a10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:57.136541 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136381    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50954522-828f-43be-9e87-b281ad092a10-istiod-ca-cert" pod "50954522-828f-43be-9e87-b281ad092a10" (UID: "50954522-828f-43be-9e87-b281ad092a10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:57.136874 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136841    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50954522-828f-43be-9e87-b281ad092a10-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:57.136874 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136864    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:57.136874 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136878    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:57.136874 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136895    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:57.136874 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.136910    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:57.138323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.138294    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-kube-api-access-l48sq" pod "50954522-828f-43be-9e87-b281ad092a10" (UID: "50954522-828f-43be-9e87-b281ad092a10"). InnerVolumeSpecName "kube-api-access-l48sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:57.138431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.138320    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-istio-token" pod "50954522-828f-43be-9e87-b281ad092a10" (UID: "50954522-828f-43be-9e87-b281ad092a10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:57.138431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.138296    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50954522-828f-43be-9e87-b281ad092a10-istio-podinfo" pod "50954522-828f-43be-9e87-b281ad092a10" (UID: "50954522-828f-43be-9e87-b281ad092a10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:13:57.138553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.138540    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-envoy" pod "50954522-828f-43be-9e87-b281ad092a10" (UID: "50954522-828f-43be-9e87-b281ad092a10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:57.237864 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.237830    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l48sq\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-kube-api-access-l48sq\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:57.237864 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.237865    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50954522-828f-43be-9e87-b281ad092a10-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:57.238139 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.237883    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50954522-828f-43be-9e87-b281ad092a10-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:57.238139 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:57.237896    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50954522-828f-43be-9e87-b281ad092a10-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:58.033880 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:58.033850    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"
Nov 02 02:13:58.033880 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:58.033860    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d" event={"ID":"50954522-828f-43be-9e87-b281ad092a10","Type":"ContainerDied","Data":"dbcba19113b704a9219e9c0705f5b4469a11efa7dd2c6176943ec6525744e35b"}
Nov 02 02:13:58.034473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:58.033943    4912 scope.go:122] "RemoveContainer" containerID="56db471be5f5433a711062d6a5e648d4600cf0712a348acc088da87a21fb391e"
Nov 02 02:13:58.249871 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:58.249839    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"]
Nov 02 02:13:58.255229 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:58.255201    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-xwl6d"]
Nov 02 02:13:59.230086 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:59.230027    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50954522-828f-43be-9e87-b281ad092a10" path="/var/lib/kubelet/pods/50954522-828f-43be-9e87-b281ad092a10/volumes"
Nov 02 02:13:59.738416 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:13:59.738381    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:04.569263 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:04.569226    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:06.048976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:06.048947    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"]
Nov 02 02:14:06.099198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:06.099167    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f294fb-f6vrr"]
Nov 02 02:14:07.234827 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:07.227920    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3923e7a-6361-4f74-ba31-517fb9cfc281" path="/var/lib/kubelet/pods/d3923e7a-6361-4f74-ba31-517fb9cfc281/volumes"
Nov 02 02:14:08.497120 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:08.497048    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:08.675768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:08.675741    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:09.254962 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:09.254927    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:12.138070 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:14:12.138017    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:14:13.056819 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:13.056786    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"]
Nov 02 02:14:13.106287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:13.106253    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b343b313-kdjk9"]
Nov 02 02:14:13.225773 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:13.225738    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f637d0b-3394-4af9-8237-8519e63188f4" path="/var/lib/kubelet/pods/7f637d0b-3394-4af9-8237-8519e63188f4/volumes"
Nov 02 02:14:14.032474 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:14.032438    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:18.744515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:18.744478    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:22.781149 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:22.781115    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"]
Nov 02 02:14:22.781575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:22.781317    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk" podUID="e50d6d24-b770-400d-bf1b-c3ceade8a35c" containerName="maas-api" containerID="cri-o://229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f" gracePeriod=30
Nov 02 02:14:22.912223 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:22.912186    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:23.066535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.066503    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:14:23.216697 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.216663    4912 generic.go:356] "Generic (PLEG): container finished" podID="e50d6d24-b770-400d-bf1b-c3ceade8a35c" containerID="229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f" exitCode=0
Nov 02 02:14:23.216896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.216738    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"
Nov 02 02:14:23.216896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.216787    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk" event={"ID":"e50d6d24-b770-400d-bf1b-c3ceade8a35c","Type":"ContainerDied","Data":"229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f"}
Nov 02 02:14:23.216896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.216843    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk" event={"ID":"e50d6d24-b770-400d-bf1b-c3ceade8a35c","Type":"ContainerDied","Data":"fff56ec4486a29d82e0a3172f31f377979eb3e5ba92baa71f6e00341874ad602"}
Nov 02 02:14:23.216896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.216871    4912 scope.go:122] "RemoveContainer" containerID="229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f"
Nov 02 02:14:23.228853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.228823    4912 scope.go:122] "RemoveContainer" containerID="229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f"
Nov 02 02:14:23.229254 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:14:23.229225    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f\": container with ID starting with 229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f not found: ID does not exist" containerID="229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f"
Nov 02 02:14:23.229339 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.229265    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f"} err="failed to get container status \"229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f\": rpc error: code = NotFound desc = could not find container \"229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f\": container with ID starting with 229094c7f834df80238af6831d14b36354dff28b6726557e6bfc2bd6269d2d1f not found: ID does not exist"
Nov 02 02:14:23.234538 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.234516    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls\") pod \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") "
Nov 02 02:14:23.234643 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.234563    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e50d6d24-b770-400d-bf1b-c3ceade8a35c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50d6d24-b770-400d-bf1b-c3ceade8a35c-openshift-service-ca\") pod \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") "
Nov 02 02:14:23.234643 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.234604    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-metrics-certs\") pod \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") "
Nov 02 02:14:23.234787 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.234775    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50d6d24-b770-400d-bf1b-c3ceade8a35c-kube-api-access-6xq8v\" (UniqueName: \"kubernetes.io/projected/e50d6d24-b770-400d-bf1b-c3ceade8a35c-kube-api-access-6xq8v\") pod \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\" (UID: \"e50d6d24-b770-400d-bf1b-c3ceade8a35c\") "
Nov 02 02:14:23.234958 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.234933    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50d6d24-b770-400d-bf1b-c3ceade8a35c-openshift-service-ca" pod "e50d6d24-b770-400d-bf1b-c3ceade8a35c" (UID: "e50d6d24-b770-400d-bf1b-c3ceade8a35c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:23.235147 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.235129    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50d6d24-b770-400d-bf1b-c3ceade8a35c-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:23.236721 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.236701    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls" pod "e50d6d24-b770-400d-bf1b-c3ceade8a35c" (UID: "e50d6d24-b770-400d-bf1b-c3ceade8a35c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:23.236878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.236855    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-metrics-certs" pod "e50d6d24-b770-400d-bf1b-c3ceade8a35c" (UID: "e50d6d24-b770-400d-bf1b-c3ceade8a35c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:23.236970 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.236952    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50d6d24-b770-400d-bf1b-c3ceade8a35c-kube-api-access-6xq8v" pod "e50d6d24-b770-400d-bf1b-c3ceade8a35c" (UID: "e50d6d24-b770-400d-bf1b-c3ceade8a35c"). InnerVolumeSpecName "kube-api-access-6xq8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:23.336014 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.335928    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xq8v\" (UniqueName: \"kubernetes.io/projected/e50d6d24-b770-400d-bf1b-c3ceade8a35c-kube-api-access-6xq8v\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:23.336014 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.335958    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:23.336014 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.335967    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e50d6d24-b770-400d-bf1b-c3ceade8a35c-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:23.638284 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.638244    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"]
Nov 02 02:14:23.641693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:23.641666    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76837e-5f9bd6789d-fghpk"]
Nov 02 02:14:24.414123 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:24.414086    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:24.763534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:24.763433    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:24.941816 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:24.941774    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:25.225765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:25.225733    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e50d6d24-b770-400d-bf1b-c3ceade8a35c" path="/var/lib/kubelet/pods/e50d6d24-b770-400d-bf1b-c3ceade8a35c/volumes"
Nov 02 02:14:25.473281 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:25.473248    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:29.651723 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:29.651630    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:30.050091 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:30.049968    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"]
Nov 02 02:14:30.101306 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:30.101273    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4355d1e2-pjlws"]
Nov 02 02:14:30.105766 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:30.105740    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"]
Nov 02 02:14:30.108510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:30.108486    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbfb6b19-trsw4"]
Nov 02 02:14:31.232028 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:31.231995    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decdd346-1f3e-445a-b939-e823fe3890af" path="/var/lib/kubelet/pods/decdd346-1f3e-445a-b939-e823fe3890af/volumes"
Nov 02 02:14:31.233273 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:31.233248    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae483f0-4869-4652-8d36-74f0cfb7006d" path="/var/lib/kubelet/pods/eae483f0-4869-4652-8d36-74f0cfb7006d/volumes"
Nov 02 02:14:31.653569 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:31.653533    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:31.836874 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:31.836837    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:32.016504 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:32.016407    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:32.484755 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:32.484719    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:34.643376 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:34.643341    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:37.410602 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:37.410568    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"]
Nov 02 02:14:37.538560 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:37.538528    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:39.749762 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:39.749722    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:42.139337 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:14:42.139301    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:14:44.539175 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:44.539142    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:44.887442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:44.887405    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:45.110554 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:45.110516    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:49.289799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:49.289757    4912 scope.go:122] "RemoveContainer" containerID="b4c11f5fecd6a00e0293ae8931bfeff248d79b7988444625c67b7a5392c5aef3"
Nov 02 02:14:49.300357 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:49.300335    4912 scope.go:122] "RemoveContainer" containerID="53ffb8f536c83c1a1c37b2b09c5b6632b1d9cff79b744714fa2575051a5104da"
Nov 02 02:14:49.309758 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:49.309738    4912 scope.go:122] "RemoveContainer" containerID="b9471caca5fe41c4f7f4d06dcd52813e417b0aa111fdaed11589ba791c0158ee"
Nov 02 02:14:49.319402 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:49.319380    4912 scope.go:122] "RemoveContainer" containerID="080c32ac169b20aaf4de877148517c6c5342699c42aaf343b382894355a7f06f"
Nov 02 02:14:49.354694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:49.354668    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:52.431728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.431692    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main" containerID="cri-o://86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737" gracePeriod=45
Nov 02 02:14:52.639888 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.639847    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:52.708955 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.708931    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 02:14:52.859316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.859286    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tmp-dir\") pod \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") "
Nov 02 02:14:52.859515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.859342    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-home\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-home\") pod \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") "
Nov 02 02:14:52.859515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.859384    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-model-cache\") pod \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") "
Nov 02 02:14:52.859515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.859443    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tls-certs\" (UniqueName: \"kubernetes.io/secret/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tls-certs\") pod \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") "
Nov 02 02:14:52.859515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.859468    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-dshm\") pod \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") "
Nov 02 02:14:52.859748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.859533    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff3e499-a2c2-4e76-a04b-7f054aef3db7-kube-api-access-8l9l5\" (UniqueName: \"kubernetes.io/projected/eff3e499-a2c2-4e76-a04b-7f054aef3db7-kube-api-access-8l9l5\") pod \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\" (UID: \"eff3e499-a2c2-4e76-a04b-7f054aef3db7\") "
Nov 02 02:14:52.859748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.859565    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tmp-dir" pod "eff3e499-a2c2-4e76-a04b-7f054aef3db7" (UID: "eff3e499-a2c2-4e76-a04b-7f054aef3db7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:52.859748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.859611    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-home" pod "eff3e499-a2c2-4e76-a04b-7f054aef3db7" (UID: "eff3e499-a2c2-4e76-a04b-7f054aef3db7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:52.859748 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.859711    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-model-cache" pod "eff3e499-a2c2-4e76-a04b-7f054aef3db7" (UID: "eff3e499-a2c2-4e76-a04b-7f054aef3db7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:52.860418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.860265    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.860575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.860422    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.860575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.860438    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.861958 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.861926    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-dshm" pod "eff3e499-a2c2-4e76-a04b-7f054aef3db7" (UID: "eff3e499-a2c2-4e76-a04b-7f054aef3db7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:52.862115 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.862046    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff3e499-a2c2-4e76-a04b-7f054aef3db7-kube-api-access-8l9l5" pod "eff3e499-a2c2-4e76-a04b-7f054aef3db7" (UID: "eff3e499-a2c2-4e76-a04b-7f054aef3db7"). InnerVolumeSpecName "kube-api-access-8l9l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:52.862115 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.862089    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tls-certs" pod "eff3e499-a2c2-4e76-a04b-7f054aef3db7" (UID: "eff3e499-a2c2-4e76-a04b-7f054aef3db7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:52.961872 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.961790    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff3e499-a2c2-4e76-a04b-7f054aef3db7-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.961872 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.961826    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff3e499-a2c2-4e76-a04b-7f054aef3db7-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.961872 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:52.961841    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l9l5\" (UniqueName: \"kubernetes.io/projected/eff3e499-a2c2-4e76-a04b-7f054aef3db7-kube-api-access-8l9l5\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:53.464381 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:53.464347    4912 generic.go:356] "Generic (PLEG): container finished" podID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerID="86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737" exitCode=0
Nov 02 02:14:53.464840 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:53.464471    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"
Nov 02 02:14:53.464840 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:53.464499    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" event={"ID":"eff3e499-a2c2-4e76-a04b-7f054aef3db7","Type":"ContainerDied","Data":"86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737"}
Nov 02 02:14:53.464840 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:53.464552    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8" event={"ID":"eff3e499-a2c2-4e76-a04b-7f054aef3db7","Type":"ContainerDied","Data":"bb90ca5c0a3b7bed7f325d86611464d4d354dda620bb383f65864fd136da8fbd"}
Nov 02 02:14:53.464840 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:53.464579    4912 scope.go:122] "RemoveContainer" containerID="86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737"
Nov 02 02:14:53.475166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:53.475147    4912 scope.go:122] "RemoveContainer" containerID="86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737"
Nov 02 02:14:53.475475 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:14:53.475452    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737\": container with ID starting with 86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737 not found: ID does not exist" containerID="86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737"
Nov 02 02:14:53.475573 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:53.475490    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737"} err="failed to get container status \"86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737\": rpc error: code = NotFound desc = could not find container \"86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737\": container with ID starting with 86437104f02806233217373eac20499d9f4d334f29a3c6814d49ee75e61c3737 not found: ID does not exist"
Nov 02 02:14:53.629175 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:53.629138    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"]
Nov 02 02:14:53.681287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:53.681243    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-w2-c62671-kserve-5b79cbf9cd-kwgp8"]
Nov 02 02:14:54.147698 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:54.147660    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:55.224692 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:55.224656    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" path="/var/lib/kubelet/pods/eff3e499-a2c2-4e76-a04b-7f054aef3db7/volumes"
Nov 02 02:14:58.315683 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.315648    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"]
Nov 02 02:14:58.316411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316389    4912 cpu_manager.go:414] "containerMap: removing container" podUID="59b93040-c359-4e41-a8ff-923846973e4a" containerName="payload-processing"
Nov 02 02:14:58.316411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316409    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b93040-c359-4e41-a8ff-923846973e4a" containerName="payload-processing"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316436    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316444    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316488    4912 cpu_manager.go:414] "containerMap: removing container" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316495    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316510    4912 cpu_manager.go:414] "containerMap: removing container" podUID="e50d6d24-b770-400d-bf1b-c3ceade8a35c" containerName="maas-api"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316517    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e50d6d24-b770-400d-bf1b-c3ceade8a35c" containerName="maas-api"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316540    4912 cpu_manager.go:414] "containerMap: removing container" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316547    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316557    4912 cpu_manager.go:414] "containerMap: removing container" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316564    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316583    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316590    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316600    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy"
Nov 02 02:14:58.316608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316607    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316620    4912 cpu_manager.go:414] "containerMap: removing container" podUID="039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" containerName="payload-pre-processing"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316627    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" containerName="payload-pre-processing"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316651    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d1ce6436-0b1f-4f8d-8e52-112531e29105" containerName="maas-api"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316659    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ce6436-0b1f-4f8d-8e52-112531e29105" containerName="maas-api"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316672    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316679    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316704    4912 cpu_manager.go:414] "containerMap: removing container" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316711    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316840    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a49295-57be-4bb4-9148-5bde151f6ecb" containerName="main"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316851    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff3e499-a2c2-4e76-a04b-7f054aef3db7" containerName="main"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316867    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c07e88-8d03-43bb-b28f-e3ef7490775b" containerName="main"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316876    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c0dd23-fea9-49e6-af8c-4f67d31d4eb4" containerName="istio-proxy"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316885    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="039a37fd-6287-4fb0-b587-dc6fe5bcd7f2" containerName="payload-pre-processing"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316897    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ce6436-0b1f-4f8d-8e52-112531e29105" containerName="maas-api"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316907    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6995d05c-1099-418f-9905-c65248d5c660" containerName="istio-proxy"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316917    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316927    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="569172aa-26cb-46aa-8a18-d46f3c7630c6" containerName="main"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316935    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b93040-c359-4e41-a8ff-923846973e4a" containerName="payload-processing"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316948    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19ab8da-5870-43b6-be66-998a9c154b93" containerName="istio-proxy"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316959    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy"
Nov 02 02:14:58.317316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.316968    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="e50d6d24-b770-400d-bf1b-c3ceade8a35c" containerName="maas-api"
Nov 02 02:14:58.334550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.334507    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.383449 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.383417    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"]
Nov 02 02:14:58.428691 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.428657    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:14:58.428856 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.428833    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:14:58.431348 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.431330    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f288de4-2jbbf\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.431401 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.431369    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgql\" (UniqueName: \"kubernetes.io/projected/5f4daede-23e0-4199-834d-1a0ecf9cb72c-kube-api-access-grgql\") pod \"maas-api-revoke-sub-8f288de4-2jbbf\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.431455 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.431444    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f288de4-2jbbf\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.532196 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.532164    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgql\" (UniqueName: \"kubernetes.io/projected/5f4daede-23e0-4199-834d-1a0ecf9cb72c-kube-api-access-grgql\") pod \"maas-api-revoke-sub-8f288de4-2jbbf\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.532390 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.532261    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f288de4-2jbbf\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.532465 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.532411    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f288de4-2jbbf\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.532939 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.532911    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f288de4-2jbbf\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.535090 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.535042    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f288de4-2jbbf\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.593728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.593662    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgql\" (UniqueName: \"kubernetes.io/projected/5f4daede-23e0-4199-834d-1a0ecf9cb72c-kube-api-access-grgql\") pod \"maas-api-revoke-sub-8f288de4-2jbbf\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:58.648331 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:58.648290    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:14:59.047913 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:59.047881    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"]
Nov 02 02:14:59.048835 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:14:59.048810    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4daede_23e0_4199_834d_1a0ecf9cb72c.slice/crio-ece2292598dca3b41aaca86307da3c55881e73594945d409817c1398b912dd29 WatchSource:0}: Error finding container ece2292598dca3b41aaca86307da3c55881e73594945d409817c1398b912dd29: Status 404 returned error can't find the container with id ece2292598dca3b41aaca86307da3c55881e73594945d409817c1398b912dd29
Nov 02 02:14:59.158936 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:59.158902    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:14:59.508556 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:59.508514    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf" event={"ID":"5f4daede-23e0-4199-834d-1a0ecf9cb72c","Type":"ContainerStarted","Data":"e7e9d63b502c7b619c20b2b390817343dbb1d12cff66c779e8a65f7a2b892991"}
Nov 02 02:14:59.509127 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:59.508590    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf" event={"ID":"5f4daede-23e0-4199-834d-1a0ecf9cb72c","Type":"ContainerStarted","Data":"ece2292598dca3b41aaca86307da3c55881e73594945d409817c1398b912dd29"}
Nov 02 02:14:59.626334 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:14:59.626280    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf" podStartSLOduration=1.626263274 podStartE2EDuration="1.626263274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:14:59.623411743 +0000 UTC m=+2592.528877101" watchObservedRunningTime="2026-11-02 02:14:59.626263274 +0000 UTC m=+2592.531728637"
Nov 02 02:15:00.161605 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.161565    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"]
Nov 02 02:15:00.162264 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.162242    4912 cpu_manager.go:414] "containerMap: removing container" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy"
Nov 02 02:15:00.162403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.162266    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50954522-828f-43be-9e87-b281ad092a10" containerName="istio-proxy"
Nov 02 02:15:00.165552 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.165493    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"]
Nov 02 02:15:00.165682 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.165628    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:00.169007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.168988    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:00.266009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.263342    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"]
Nov 02 02:15:00.268619 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.267483    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"]
Nov 02 02:15:00.351989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.351956    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk\" (UID: \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:00.351989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.351994    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77g4w\" (UniqueName: \"kubernetes.io/projected/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-kube-api-access-77g4w\") pod \"maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk\" (UID: \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:00.352254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.352162    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84lp4\" (UniqueName: \"kubernetes.io/projected/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-kube-api-access-84lp4\") pod \"maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5\" (UID: \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:00.352254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.352209    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5\" (UID: \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:00.453247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.453165    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk\" (UID: \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:00.453247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.453199    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77g4w\" (UniqueName: \"kubernetes.io/projected/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-kube-api-access-77g4w\") pod \"maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk\" (UID: \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:00.453503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.453250    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lp4\" (UniqueName: \"kubernetes.io/projected/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-kube-api-access-84lp4\") pod \"maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5\" (UID: \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:00.453503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.453287    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5\" (UID: \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:00.453781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.453761    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk\" (UID: \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:00.453957 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.453939    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5\" (UID: \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:00.516906 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.516867    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77g4w\" (UniqueName: \"kubernetes.io/projected/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-kube-api-access-77g4w\") pod \"maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk\" (UID: \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:00.517311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.517162    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lp4\" (UniqueName: \"kubernetes.io/projected/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-kube-api-access-84lp4\") pod \"maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5\" (UID: \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:00.519365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.519339    4912 generic.go:356] "Generic (PLEG): container finished" podID="5f4daede-23e0-4199-834d-1a0ecf9cb72c" containerID="e7e9d63b502c7b619c20b2b390817343dbb1d12cff66c779e8a65f7a2b892991" exitCode=0
Nov 02 02:15:00.519485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.519414    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf" event={"ID":"5f4daede-23e0-4199-834d-1a0ecf9cb72c","Type":"ContainerDied","Data":"e7e9d63b502c7b619c20b2b390817343dbb1d12cff66c779e8a65f7a2b892991"}
Nov 02 02:15:00.780484 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.780383    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:00.787556 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:00.787527    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:01.174041 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.174013    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"]
Nov 02 02:15:01.174657 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:15:01.174627    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd75739_b39a_423a_8ddd_9a01cf4a93cf.slice/crio-d7792945ccc262576f62583dd68f72e79a9270c145c456e02f9cc03ec4b34234 WatchSource:0}: Error finding container d7792945ccc262576f62583dd68f72e79a9270c145c456e02f9cc03ec4b34234: Status 404 returned error can't find the container with id d7792945ccc262576f62583dd68f72e79a9270c145c456e02f9cc03ec4b34234
Nov 02 02:15:01.223636 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:15:01.223605    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c8faa9_9fc5_402e_b39b_8dc2c86c2b4f.slice/crio-c1b3a8961ba42124d9d7072db6c39f9bd6301a612ebf7b6d7a110333093025b7 WatchSource:0}: Error finding container c1b3a8961ba42124d9d7072db6c39f9bd6301a612ebf7b6d7a110333093025b7: Status 404 returned error can't find the container with id c1b3a8961ba42124d9d7072db6c39f9bd6301a612ebf7b6d7a110333093025b7
Nov 02 02:15:01.227847 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.227826    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"]
Nov 02 02:15:01.527530 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.527484    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" event={"ID":"8cd75739-b39a-423a-8ddd-9a01cf4a93cf","Type":"ContainerStarted","Data":"d7792945ccc262576f62583dd68f72e79a9270c145c456e02f9cc03ec4b34234"}
Nov 02 02:15:01.528819 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.528794    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" event={"ID":"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f","Type":"ContainerStarted","Data":"c1b3a8961ba42124d9d7072db6c39f9bd6301a612ebf7b6d7a110333093025b7"}
Nov 02 02:15:01.675591 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.675567    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:15:01.867475 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.867430    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-xks-ca\") pod \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") "
Nov 02 02:15:01.867709 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.867513    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-openshift-ca\") pod \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") "
Nov 02 02:15:01.867709 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.867558    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4daede-23e0-4199-834d-1a0ecf9cb72c-kube-api-access-grgql\" (UniqueName: \"kubernetes.io/projected/5f4daede-23e0-4199-834d-1a0ecf9cb72c-kube-api-access-grgql\") pod \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\" (UID: \"5f4daede-23e0-4199-834d-1a0ecf9cb72c\") "
Nov 02 02:15:01.867931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.867904    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-openshift-ca" pod "5f4daede-23e0-4199-834d-1a0ecf9cb72c" (UID: "5f4daede-23e0-4199-834d-1a0ecf9cb72c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:01.868164 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.868132    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:01.869578 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.869557    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-xks-ca" pod "5f4daede-23e0-4199-834d-1a0ecf9cb72c" (UID: "5f4daede-23e0-4199-834d-1a0ecf9cb72c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:15:01.869923 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.869893    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4daede-23e0-4199-834d-1a0ecf9cb72c-kube-api-access-grgql" pod "5f4daede-23e0-4199-834d-1a0ecf9cb72c" (UID: "5f4daede-23e0-4199-834d-1a0ecf9cb72c"). InnerVolumeSpecName "kube-api-access-grgql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:01.968785 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.968751    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grgql\" (UniqueName: \"kubernetes.io/projected/5f4daede-23e0-4199-834d-1a0ecf9cb72c-kube-api-access-grgql\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:01.968785 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:01.968784    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4daede-23e0-4199-834d-1a0ecf9cb72c-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:02.537543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:02.537454    4912 generic.go:356] "Generic (PLEG): container finished" podID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerID="9d9666c6cb7c9b2d9d9462bc2d1488ddc00edd1c2cd23a87277b74f18d1d0afc" exitCode=60
Nov 02 02:15:02.538198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:02.537535    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" event={"ID":"8cd75739-b39a-423a-8ddd-9a01cf4a93cf","Type":"ContainerDied","Data":"9d9666c6cb7c9b2d9d9462bc2d1488ddc00edd1c2cd23a87277b74f18d1d0afc"}
Nov 02 02:15:02.538198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:02.537817    4912 scope.go:122] "RemoveContainer" containerID="9d9666c6cb7c9b2d9d9462bc2d1488ddc00edd1c2cd23a87277b74f18d1d0afc"
Nov 02 02:15:02.539519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:02.539490    4912 generic.go:356] "Generic (PLEG): container finished" podID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerID="6c0205b8b1f957ee02d59e912e70964bc18ca04d88c71a2b4521c0ec6d23b326" exitCode=60
Nov 02 02:15:02.539607 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:02.539581    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" event={"ID":"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f","Type":"ContainerDied","Data":"6c0205b8b1f957ee02d59e912e70964bc18ca04d88c71a2b4521c0ec6d23b326"}
Nov 02 02:15:02.539842 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:02.539826    4912 scope.go:122] "RemoveContainer" containerID="6c0205b8b1f957ee02d59e912e70964bc18ca04d88c71a2b4521c0ec6d23b326"
Nov 02 02:15:02.541311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:02.541239    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"
Nov 02 02:15:02.541311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:02.541253    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf" event={"ID":"5f4daede-23e0-4199-834d-1a0ecf9cb72c","Type":"ContainerDied","Data":"ece2292598dca3b41aaca86307da3c55881e73594945d409817c1398b912dd29"}
Nov 02 02:15:02.541311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:02.541289    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ece2292598dca3b41aaca86307da3c55881e73594945d409817c1398b912dd29"
Nov 02 02:15:03.552132 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:03.552095    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" event={"ID":"8cd75739-b39a-423a-8ddd-9a01cf4a93cf","Type":"ContainerStarted","Data":"c0259e75301be527a1c708470be913170a47351e09ba3368f594858ef4f29682"}
Nov 02 02:15:03.554544 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:03.554517    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" event={"ID":"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f","Type":"ContainerDied","Data":"92608e3b1f4200ff036f670ade0431df9eb688f5fe56920b80b522b3b259e273"}
Nov 02 02:15:03.554682 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:03.554578    4912 scope.go:122] "RemoveContainer" containerID="6c0205b8b1f957ee02d59e912e70964bc18ca04d88c71a2b4521c0ec6d23b326"
Nov 02 02:15:03.554744 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:03.554704    4912 generic.go:356] "Generic (PLEG): container finished" podID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerID="92608e3b1f4200ff036f670ade0431df9eb688f5fe56920b80b522b3b259e273" exitCode=60
Nov 02 02:15:03.554834 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:03.554816    4912 scope.go:122] "RemoveContainer" containerID="92608e3b1f4200ff036f670ade0431df9eb688f5fe56920b80b522b3b259e273"
Nov 02 02:15:03.555159 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:03.555130    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5_odh-ai-gateway-infra(68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f"
Nov 02 02:15:03.668717 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:03.668662    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" podStartSLOduration=3.668641393 podStartE2EDuration="3.668641393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:15:03.66786276 +0000 UTC m=+2596.573328117" watchObservedRunningTime="2026-11-02 02:15:03.668641393 +0000 UTC m=+2596.574106750"
Nov 02 02:15:03.954263 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:03.954169    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:04.564621 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:04.564587    4912 generic.go:356] "Generic (PLEG): container finished" podID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerID="c0259e75301be527a1c708470be913170a47351e09ba3368f594858ef4f29682" exitCode=60
Nov 02 02:15:04.565154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:04.564678    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" event={"ID":"8cd75739-b39a-423a-8ddd-9a01cf4a93cf","Type":"ContainerDied","Data":"c0259e75301be527a1c708470be913170a47351e09ba3368f594858ef4f29682"}
Nov 02 02:15:04.565154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:04.564777    4912 scope.go:122] "RemoveContainer" containerID="9d9666c6cb7c9b2d9d9462bc2d1488ddc00edd1c2cd23a87277b74f18d1d0afc"
Nov 02 02:15:04.565154 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:04.564980    4912 scope.go:122] "RemoveContainer" containerID="c0259e75301be527a1c708470be913170a47351e09ba3368f594858ef4f29682"
Nov 02 02:15:04.565342 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:04.565312    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk_odh-ai-gateway-infra(8cd75739-b39a-423a-8ddd-9a01cf4a93cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf"
Nov 02 02:15:04.566940 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:04.566921    4912 scope.go:122] "RemoveContainer" containerID="92608e3b1f4200ff036f670ade0431df9eb688f5fe56920b80b522b3b259e273"
Nov 02 02:15:04.567223 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:04.567203    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5_odh-ai-gateway-infra(68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f"
Nov 02 02:15:05.576892 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:05.576867    4912 scope.go:122] "RemoveContainer" containerID="c0259e75301be527a1c708470be913170a47351e09ba3368f594858ef4f29682"
Nov 02 02:15:05.577362 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:05.577161    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk_odh-ai-gateway-infra(8cd75739-b39a-423a-8ddd-9a01cf4a93cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf"
Nov 02 02:15:08.747031 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:08.746994    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:12.143641 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:12.143597    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:15:13.544169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:13.544132    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:14.218346 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:14.218314    4912 scope.go:122] "RemoveContainer" containerID="c0259e75301be527a1c708470be913170a47351e09ba3368f594858ef4f29682"
Nov 02 02:15:14.687994 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:14.687947    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:14.870595 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:14.870561    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:15.093338 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:15.093130    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:15.218268 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:15.218237    4912 scope.go:122] "RemoveContainer" containerID="92608e3b1f4200ff036f670ade0431df9eb688f5fe56920b80b522b3b259e273"
Nov 02 02:15:15.665334 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:15.665295    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" event={"ID":"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f","Type":"ContainerStarted","Data":"a8f6e39edfbc8696aa49741ecc28118aa0cc51d1670eaf800b15c2708453cecb"}
Nov 02 02:15:15.668567 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:15.668533    4912 generic.go:356] "Generic (PLEG): container finished" podID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerID="a755f1c48a0794bb266e58364514aa20dc8a246431948d13ba32cd2d9ba4313e" exitCode=60
Nov 02 02:15:15.668711 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:15.668603    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" event={"ID":"8cd75739-b39a-423a-8ddd-9a01cf4a93cf","Type":"ContainerDied","Data":"a755f1c48a0794bb266e58364514aa20dc8a246431948d13ba32cd2d9ba4313e"}
Nov 02 02:15:15.668711 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:15.668685    4912 scope.go:122] "RemoveContainer" containerID="c0259e75301be527a1c708470be913170a47351e09ba3368f594858ef4f29682"
Nov 02 02:15:15.669175 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:15.669148    4912 scope.go:122] "RemoveContainer" containerID="a755f1c48a0794bb266e58364514aa20dc8a246431948d13ba32cd2d9ba4313e"
Nov 02 02:15:15.669597 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:15.669569    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk_odh-ai-gateway-infra(8cd75739-b39a-423a-8ddd-9a01cf4a93cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf"
Nov 02 02:15:15.781961 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:15.781843    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" podStartSLOduration=15.781827936 podStartE2EDuration="15.781827936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:15:15.780248106 +0000 UTC m=+2608.685713465" watchObservedRunningTime="2026-11-02 02:15:15.781827936 +0000 UTC m=+2608.687293289"
Nov 02 02:15:16.680335 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:16.680289    4912 generic.go:356] "Generic (PLEG): container finished" podID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerID="a8f6e39edfbc8696aa49741ecc28118aa0cc51d1670eaf800b15c2708453cecb" exitCode=60
Nov 02 02:15:16.680556 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:16.680376    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" event={"ID":"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f","Type":"ContainerDied","Data":"a8f6e39edfbc8696aa49741ecc28118aa0cc51d1670eaf800b15c2708453cecb"}
Nov 02 02:15:16.680556 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:16.680444    4912 scope.go:122] "RemoveContainer" containerID="92608e3b1f4200ff036f670ade0431df9eb688f5fe56920b80b522b3b259e273"
Nov 02 02:15:16.680862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:16.680835    4912 scope.go:122] "RemoveContainer" containerID="a8f6e39edfbc8696aa49741ecc28118aa0cc51d1670eaf800b15c2708453cecb"
Nov 02 02:15:16.681214 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:16.681192    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5_odh-ai-gateway-infra(68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f"
Nov 02 02:15:16.790723 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:16.790685    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"]
Nov 02 02:15:16.905068 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:16.905014    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"]
Nov 02 02:15:17.070448 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.070420    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:17.138622 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.138583    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-maas-api-service-ca\") pod \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\" (UID: \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\") "
Nov 02 02:15:17.138820 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.138692    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-kube-api-access-77g4w\" (UniqueName: \"kubernetes.io/projected/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-kube-api-access-77g4w\") pod \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\" (UID: \"8cd75739-b39a-423a-8ddd-9a01cf4a93cf\") "
Nov 02 02:15:17.139045 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.139021    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-maas-api-service-ca" pod "8cd75739-b39a-423a-8ddd-9a01cf4a93cf" (UID: "8cd75739-b39a-423a-8ddd-9a01cf4a93cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:17.139210 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.139194    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:17.141173 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.141133    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-kube-api-access-77g4w" pod "8cd75739-b39a-423a-8ddd-9a01cf4a93cf" (UID: "8cd75739-b39a-423a-8ddd-9a01cf4a93cf"). InnerVolumeSpecName "kube-api-access-77g4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:17.240690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.240660    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77g4w\" (UniqueName: \"kubernetes.io/projected/8cd75739-b39a-423a-8ddd-9a01cf4a93cf-kube-api-access-77g4w\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:17.693167 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.693135    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"
Nov 02 02:15:17.693167 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.693165    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk" event={"ID":"8cd75739-b39a-423a-8ddd-9a01cf4a93cf","Type":"ContainerDied","Data":"d7792945ccc262576f62583dd68f72e79a9270c145c456e02f9cc03ec4b34234"}
Nov 02 02:15:17.693628 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.693233    4912 scope.go:122] "RemoveContainer" containerID="a755f1c48a0794bb266e58364514aa20dc8a246431948d13ba32cd2d9ba4313e"
Nov 02 02:15:17.865607 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.865580    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"]
Nov 02 02:15:17.869716 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.869688    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4211c0-29893095-l2czk"]
Nov 02 02:15:17.883191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.883172    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:17.947740 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.947667    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-kube-api-access-84lp4\" (UniqueName: \"kubernetes.io/projected/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-kube-api-access-84lp4\") pod \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\" (UID: \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\") "
Nov 02 02:15:17.947862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.947845    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-maas-api-service-ca\") pod \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\" (UID: \"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f\") "
Nov 02 02:15:17.948236 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.948213    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-maas-api-service-ca" pod "68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" (UID: "68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:17.950087 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:17.950050    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-kube-api-access-84lp4" pod "68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" (UID: "68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f"). InnerVolumeSpecName "kube-api-access-84lp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:18.049714 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:18.049680    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:18.049714 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:18.049716    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84lp4\" (UniqueName: \"kubernetes.io/projected/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f-kube-api-access-84lp4\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:18.179594 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:18.179560    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:18.366309 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:18.366270    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:18.706363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:18.706265    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5" event={"ID":"68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f","Type":"ContainerDied","Data":"c1b3a8961ba42124d9d7072db6c39f9bd6301a612ebf7b6d7a110333093025b7"}
Nov 02 02:15:18.706363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:18.706346    4912 scope.go:122] "RemoveContainer" containerID="a8f6e39edfbc8696aa49741ecc28118aa0cc51d1670eaf800b15c2708453cecb"
Nov 02 02:15:18.706593 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:18.706282    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"
Nov 02 02:15:18.869158 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:18.869123    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"]
Nov 02 02:15:18.875325 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:18.875298    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c942d-29893095-wcpl5"]
Nov 02 02:15:19.225690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:19.225655    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" path="/var/lib/kubelet/pods/68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f/volumes"
Nov 02 02:15:19.226110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:19.226096    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" path="/var/lib/kubelet/pods/8cd75739-b39a-423a-8ddd-9a01cf4a93cf/volumes"
Nov 02 02:15:21.045454 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:21.045423    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:21.219387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:21.219334    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:22.515367 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:22.515335    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:24.120789 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:24.120751    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:24.346866 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:24.346830    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:24.991431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:24.991391    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:29.333986 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:29.333947    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:31.040452 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:31.040411    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:31.222521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:31.222481    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:31.359020 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:31.358933    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"]
Nov 02 02:15:31.359274 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:31.359244    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy" containerID="cri-o://8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115" gracePeriod=30
Nov 02 02:15:31.403877 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:31.403836    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:34.048845 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:34.048812    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:36.626651 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.626619    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 02:15:36.763416 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763323    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4633de5-ffcd-4056-8677-cd5426f8a643-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4633de5-ffcd-4056-8677-cd5426f8a643-istiod-ca-cert\") pod \"d4633de5-ffcd-4056-8677-cd5426f8a643\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") "
Nov 02 02:15:36.763416 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763375    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-podinfo\") pod \"d4633de5-ffcd-4056-8677-cd5426f8a643\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") "
Nov 02 02:15:36.763416 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763414    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-kube-api-access-cnxp8\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-kube-api-access-cnxp8\") pod \"d4633de5-ffcd-4056-8677-cd5426f8a643\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") "
Nov 02 02:15:36.763737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763437    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-token\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-token\") pod \"d4633de5-ffcd-4056-8677-cd5426f8a643\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") "
Nov 02 02:15:36.763737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763468    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-envoy\") pod \"d4633de5-ffcd-4056-8677-cd5426f8a643\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") "
Nov 02 02:15:36.763737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763506    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-data\") pod \"d4633de5-ffcd-4056-8677-cd5426f8a643\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") "
Nov 02 02:15:36.763737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763579    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-certs\") pod \"d4633de5-ffcd-4056-8677-cd5426f8a643\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") "
Nov 02 02:15:36.763737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763673    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-credential-socket\") pod \"d4633de5-ffcd-4056-8677-cd5426f8a643\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") "
Nov 02 02:15:36.763737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763719    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-socket\") pod \"d4633de5-ffcd-4056-8677-cd5426f8a643\" (UID: \"d4633de5-ffcd-4056-8677-cd5426f8a643\") "
Nov 02 02:15:36.764108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763919    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-data" pod "d4633de5-ffcd-4056-8677-cd5426f8a643" (UID: "d4633de5-ffcd-4056-8677-cd5426f8a643"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:15:36.764108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.763832    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4633de5-ffcd-4056-8677-cd5426f8a643-istiod-ca-cert" pod "d4633de5-ffcd-4056-8677-cd5426f8a643" (UID: "d4633de5-ffcd-4056-8677-cd5426f8a643"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:36.764108 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.764041    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-certs" pod "d4633de5-ffcd-4056-8677-cd5426f8a643" (UID: "d4633de5-ffcd-4056-8677-cd5426f8a643"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:15:36.764340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.764194    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-credential-socket" pod "d4633de5-ffcd-4056-8677-cd5426f8a643" (UID: "d4633de5-ffcd-4056-8677-cd5426f8a643"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:15:36.764340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.764211    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-socket" pod "d4633de5-ffcd-4056-8677-cd5426f8a643" (UID: "d4633de5-ffcd-4056-8677-cd5426f8a643"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:15:36.764557 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.764537    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:36.764642 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.764561    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:36.764642 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.764577    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4633de5-ffcd-4056-8677-cd5426f8a643-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:36.764642 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.764590    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:36.764642 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.764603    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:36.766443 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.766410    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-envoy" pod "d4633de5-ffcd-4056-8677-cd5426f8a643" (UID: "d4633de5-ffcd-4056-8677-cd5426f8a643"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:15:36.766443 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.766420    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-kube-api-access-cnxp8" pod "d4633de5-ffcd-4056-8677-cd5426f8a643" (UID: "d4633de5-ffcd-4056-8677-cd5426f8a643"). InnerVolumeSpecName "kube-api-access-cnxp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:36.766668 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.766641    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-podinfo" pod "d4633de5-ffcd-4056-8677-cd5426f8a643" (UID: "d4633de5-ffcd-4056-8677-cd5426f8a643"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:15:36.766787 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.766728    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-token" pod "d4633de5-ffcd-4056-8677-cd5426f8a643" (UID: "d4633de5-ffcd-4056-8677-cd5426f8a643"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:36.846190 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.846154    4912 generic.go:356] "Generic (PLEG): container finished" podID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerID="8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115" exitCode=0
Nov 02 02:15:36.846392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.846229    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"
Nov 02 02:15:36.846392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.846254    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" event={"ID":"d4633de5-ffcd-4056-8677-cd5426f8a643","Type":"ContainerDied","Data":"8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115"}
Nov 02 02:15:36.846392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.846317    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9" event={"ID":"d4633de5-ffcd-4056-8677-cd5426f8a643","Type":"ContainerDied","Data":"bfdb00b4769ae1348b9307c720c464cb30fa40bede2a99e885d23c235f35a092"}
Nov 02 02:15:36.846392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.846352    4912 scope.go:122] "RemoveContainer" containerID="8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115"
Nov 02 02:15:36.858446 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.858426    4912 scope.go:122] "RemoveContainer" containerID="8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115"
Nov 02 02:15:36.858776 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:36.858749    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115\": container with ID starting with 8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115 not found: ID does not exist" containerID="8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115"
Nov 02 02:15:36.858882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.858787    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115"} err="failed to get container status \"8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115\": rpc error: code = NotFound desc = could not find container \"8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115\": container with ID starting with 8fb4e84d343a347412a999cb7ee1a66709d7595e7b830e84df80a32ddaaa8115 not found: ID does not exist"
Nov 02 02:15:36.865446 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.865425    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:36.865532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.865448    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnxp8\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-kube-api-access-cnxp8\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:36.865532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.865459    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:36.865532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:36.865468    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4633de5-ffcd-4056-8677-cd5426f8a643-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:37.011528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:37.011483    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"]
Nov 02 02:15:37.015858 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:37.015794    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-gsgp9"]
Nov 02 02:15:37.226242 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:37.226207    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" path="/var/lib/kubelet/pods/d4633de5-ffcd-4056-8677-cd5426f8a643/volumes"
Nov 02 02:15:38.826639 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:38.826608    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:40.630002 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:40.629971    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"]
Nov 02 02:15:40.682094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:40.682014    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:42.138412 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:42.138369    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:15:43.565833 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:43.565767    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:48.350357 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:48.350317    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:49.392165 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:49.392134    4912 scope.go:122] "RemoveContainer" containerID="14a85277dbb4db219d6f80c7f3e31f27f7d354866ac8298f4ee64e3f89b8ad40"
Nov 02 02:15:49.402203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:49.402179    4912 scope.go:122] "RemoveContainer" containerID="5c7a2210c150b8e255c73ab3d9164187b6093333ea988e014fcc5a268bbc41b1"
Nov 02 02:15:49.411288 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:49.411262    4912 scope.go:122] "RemoveContainer" containerID="4a66f7bda6d21091bdc10d49a7f904e2e4f931861a45585caf21db740f940be0"
Nov 02 02:15:49.420448 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:49.420426    4912 scope.go:122] "RemoveContainer" containerID="2b800bfa25afe4666d65e5bce300094aa6b8f348032e58a6defc2da6409312c2"
Nov 02 02:15:49.429568 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:49.429551    4912 scope.go:122] "RemoveContainer" containerID="e8181a38afe3cc0909876963b00d0a509b24c6d422e400aec4f8de263fa6896c"
Nov 02 02:15:49.437957 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:49.437940    4912 scope.go:122] "RemoveContainer" containerID="b382b7f3c9b27ce65e5c0a6d747efdfa0c2e91ce8726e978bf8da3f5f8c9738c"
Nov 02 02:15:49.446485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:49.446467    4912 scope.go:122] "RemoveContainer" containerID="207867a32c98cbd44ebf3f8f9661455977be52029d5a4096b50baedc00b0a39c"
Nov 02 02:15:52.996210 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:52.996171    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:53.176841 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:53.176799    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:15:55.650678 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.650641    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585" podUID="1a4bb112-0081-4f0f-9f12-914574e2a13a" containerName="main" containerID="cri-o://6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34" gracePeriod=45
Nov 02 02:15:55.930793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.930766    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:15:55.964243 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964215    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-home\") pod \"1a4bb112-0081-4f0f-9f12-914574e2a13a\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") "
Nov 02 02:15:55.964411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964318    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-tmp-dir\") pod \"1a4bb112-0081-4f0f-9f12-914574e2a13a\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") "
Nov 02 02:15:55.964411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964382    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4bb112-0081-4f0f-9f12-914574e2a13a-kube-api-access-cwn7d\" (UniqueName: \"kubernetes.io/projected/1a4bb112-0081-4f0f-9f12-914574e2a13a-kube-api-access-cwn7d\") pod \"1a4bb112-0081-4f0f-9f12-914574e2a13a\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") "
Nov 02 02:15:55.964542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964444    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a4bb112-0081-4f0f-9f12-914574e2a13a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb112-0081-4f0f-9f12-914574e2a13a-tls-certs\") pod \"1a4bb112-0081-4f0f-9f12-914574e2a13a\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") "
Nov 02 02:15:55.964542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964466    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-dshm\") pod \"1a4bb112-0081-4f0f-9f12-914574e2a13a\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") "
Nov 02 02:15:55.964542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964502    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-home" pod "1a4bb112-0081-4f0f-9f12-914574e2a13a" (UID: "1a4bb112-0081-4f0f-9f12-914574e2a13a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:15:55.964542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964513    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-model-cache\") pod \"1a4bb112-0081-4f0f-9f12-914574e2a13a\" (UID: \"1a4bb112-0081-4f0f-9f12-914574e2a13a\") "
Nov 02 02:15:55.964738 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964606    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-tmp-dir" pod "1a4bb112-0081-4f0f-9f12-914574e2a13a" (UID: "1a4bb112-0081-4f0f-9f12-914574e2a13a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:15:55.964738 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964690    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-model-cache" pod "1a4bb112-0081-4f0f-9f12-914574e2a13a" (UID: "1a4bb112-0081-4f0f-9f12-914574e2a13a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:15:55.964960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964943    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:55.965110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964982    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:55.965110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.964995    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:55.966768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.966719    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4bb112-0081-4f0f-9f12-914574e2a13a-kube-api-access-cwn7d" pod "1a4bb112-0081-4f0f-9f12-914574e2a13a" (UID: "1a4bb112-0081-4f0f-9f12-914574e2a13a"). InnerVolumeSpecName "kube-api-access-cwn7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:55.966882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.966829    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-dshm" pod "1a4bb112-0081-4f0f-9f12-914574e2a13a" (UID: "1a4bb112-0081-4f0f-9f12-914574e2a13a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:15:55.966932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.966897    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4bb112-0081-4f0f-9f12-914574e2a13a-tls-certs" pod "1a4bb112-0081-4f0f-9f12-914574e2a13a" (UID: "1a4bb112-0081-4f0f-9f12-914574e2a13a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:15:55.984115 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.984090    4912 generic.go:356] "Generic (PLEG): container finished" podID="1a4bb112-0081-4f0f-9f12-914574e2a13a" containerID="6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34" exitCode=0
Nov 02 02:15:55.984227 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.984133    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"
Nov 02 02:15:55.984227 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.984167    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585" event={"ID":"1a4bb112-0081-4f0f-9f12-914574e2a13a","Type":"ContainerDied","Data":"6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34"}
Nov 02 02:15:55.984227 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.984210    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585" event={"ID":"1a4bb112-0081-4f0f-9f12-914574e2a13a","Type":"ContainerDied","Data":"f82691326a6841d40ed01c55d8aec7fbae4d59cd8be6fa789f4d09605cca6889"}
Nov 02 02:15:55.984335 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.984230    4912 scope.go:122] "RemoveContainer" containerID="6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34"
Nov 02 02:15:55.995302 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.995284    4912 scope.go:122] "RemoveContainer" containerID="6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34"
Nov 02 02:15:55.995587 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:15:55.995566    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34\": container with ID starting with 6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34 not found: ID does not exist" containerID="6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34"
Nov 02 02:15:55.995649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:55.995597    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34"} err="failed to get container status \"6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34\": rpc error: code = NotFound desc = could not find container \"6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34\": container with ID starting with 6c583918004cf3842e7aeab6e461dc6cca07dd0ebe1774a88f37b28a52fafb34 not found: ID does not exist"
Nov 02 02:15:56.065693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:56.065661    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4bb112-0081-4f0f-9f12-914574e2a13a-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:56.065693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:56.065691    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4bb112-0081-4f0f-9f12-914574e2a13a-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:56.065903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:56.065704    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwn7d\" (UniqueName: \"kubernetes.io/projected/1a4bb112-0081-4f0f-9f12-914574e2a13a-kube-api-access-cwn7d\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:56.158646 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:56.158562    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"]
Nov 02 02:15:56.164765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:56.164736    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1c942d/test-model-1c942d-kserve-85d795497-78585"]
Nov 02 02:15:57.225998 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:57.225901    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4bb112-0081-4f0f-9f12-914574e2a13a" path="/var/lib/kubelet/pods/1a4bb112-0081-4f0f-9f12-914574e2a13a/volumes"
Nov 02 02:15:57.853414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:15:57.853373    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:02.659575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:02.657875    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:07.450387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:07.450345    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:08.805822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:08.805778    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:12.138381 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:16:12.138347    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:16:12.468332 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:12.468243    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:17.450113 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:17.450078    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:20.799400 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:20.799361    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"]
Nov 02 02:16:20.904616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:20.904583    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"]
Nov 02 02:16:21.186430 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:21.186387    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:21.460301 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:21.460209    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:22.164342 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:22.164302    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:25.799712 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:25.799650    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq" podUID="eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" containerName="payload-processing" containerID="cri-o://2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20" gracePeriod=25
Nov 02 02:16:25.905531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:25.905491    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n" podUID="717f0211-196d-4bc5-8725-a9a6d8bae6d1" containerName="payload-pre-processing" containerID="cri-o://261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2" gracePeriod=25
Nov 02 02:16:26.970680 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:26.970642    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:31.096786 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:31.096740    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:32.615094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:32.615019    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:32.833550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:32.833508    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:33.011877 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:33.011790    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:33.274521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:33.274429    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n" podUID="717f0211-196d-4bc5-8725-a9a6d8bae6d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:16:33.275246 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:33.275219    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq" podUID="eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:16:33.564288 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:33.564205    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:37.951569 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:37.951530    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:39.409271 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:39.409229    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:39.637423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:39.637374    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:39.827089 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:39.826966    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:42.137715 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:16:42.137677    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:16:43.036230 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:43.036188    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:43.275290 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:43.275252    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq" podUID="eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:16:43.275725 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:43.275298    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n" podUID="717f0211-196d-4bc5-8725-a9a6d8bae6d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:16:46.071680 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.071616    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:16:46.167514 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.167484    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:16:46.228868 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.228836    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-plugins-config-volume\") pod \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\" (UID: \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\") "
Nov 02 02:16:46.229075 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.228904    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-kube-api-access-zspkw\" (UniqueName: \"kubernetes.io/projected/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-kube-api-access-zspkw\") pod \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\" (UID: \"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6\") "
Nov 02 02:16:46.229228 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.229205    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-plugins-config-volume" pod "eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" (UID: "eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:16:46.231280 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.231250    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-kube-api-access-zspkw" pod "eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" (UID: "eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6"). InnerVolumeSpecName "kube-api-access-zspkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:16:46.329496 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.329401    4912 generic.go:356] "Generic (PLEG): container finished" podID="717f0211-196d-4bc5-8725-a9a6d8bae6d1" containerID="261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2" exitCode=0
Nov 02 02:16:46.329693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.329528    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717f0211-196d-4bc5-8725-a9a6d8bae6d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f0211-196d-4bc5-8725-a9a6d8bae6d1-plugins-config-volume\") pod \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\" (UID: \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\") "
Nov 02 02:16:46.329693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.329620    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"
Nov 02 02:16:46.329693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.329653    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717f0211-196d-4bc5-8725-a9a6d8bae6d1-kube-api-access-kbprf\" (UniqueName: \"kubernetes.io/projected/717f0211-196d-4bc5-8725-a9a6d8bae6d1-kube-api-access-kbprf\") pod \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\" (UID: \"717f0211-196d-4bc5-8725-a9a6d8bae6d1\") "
Nov 02 02:16:46.330010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.329982    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n" event={"ID":"717f0211-196d-4bc5-8725-a9a6d8bae6d1","Type":"ContainerDied","Data":"261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2"}
Nov 02 02:16:46.330109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.330043    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n" event={"ID":"717f0211-196d-4bc5-8725-a9a6d8bae6d1","Type":"ContainerDied","Data":"a2a8935eafb32629f62453c99519f08d7f45b05def14f5680c86c3bd067bfbcf"}
Nov 02 02:16:46.330109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.330088    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zspkw\" (UniqueName: \"kubernetes.io/projected/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-kube-api-access-zspkw\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:46.330109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.330106    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:46.330296 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.330109    4912 scope.go:122] "RemoveContainer" containerID="261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2"
Nov 02 02:16:46.330661 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.330633    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717f0211-196d-4bc5-8725-a9a6d8bae6d1-plugins-config-volume" pod "717f0211-196d-4bc5-8725-a9a6d8bae6d1" (UID: "717f0211-196d-4bc5-8725-a9a6d8bae6d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:16:46.333577 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.333538    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717f0211-196d-4bc5-8725-a9a6d8bae6d1-kube-api-access-kbprf" pod "717f0211-196d-4bc5-8725-a9a6d8bae6d1" (UID: "717f0211-196d-4bc5-8725-a9a6d8bae6d1"). InnerVolumeSpecName "kube-api-access-kbprf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:16:46.346076 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.346030    4912 generic.go:356] "Generic (PLEG): container finished" podID="eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" containerID="2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20" exitCode=0
Nov 02 02:16:46.346218 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.346114    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"
Nov 02 02:16:46.346218 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.346163    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq" event={"ID":"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6","Type":"ContainerDied","Data":"2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20"}
Nov 02 02:16:46.346338 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.346237    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq" event={"ID":"eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6","Type":"ContainerDied","Data":"b17ae4326c26d685503b86834acd90db459bd763110a07b165fc2c583f97cc6c"}
Nov 02 02:16:46.355369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.355343    4912 scope.go:122] "RemoveContainer" containerID="261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2"
Nov 02 02:16:46.355673 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:16:46.355652    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2\": container with ID starting with 261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2 not found: ID does not exist" containerID="261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2"
Nov 02 02:16:46.355735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.355682    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2"} err="failed to get container status \"261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2\": rpc error: code = NotFound desc = could not find container \"261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2\": container with ID starting with 261055cd79608315c9a600ae7c3b21a87366c932c0daac34a5e860b471998ad2 not found: ID does not exist"
Nov 02 02:16:46.355735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.355702    4912 scope.go:122] "RemoveContainer" containerID="2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20"
Nov 02 02:16:46.365285 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.365268    4912 scope.go:122] "RemoveContainer" containerID="2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20"
Nov 02 02:16:46.365563 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:16:46.365542    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20\": container with ID starting with 2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20 not found: ID does not exist" containerID="2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20"
Nov 02 02:16:46.365633 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.365570    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20"} err="failed to get container status \"2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20\": rpc error: code = NotFound desc = could not find container \"2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20\": container with ID starting with 2f6f5458ea4c5c1714c031f4a72f71863da215faf1a510e9b3936a3a28087b20 not found: ID does not exist"
Nov 02 02:16:46.431383 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.431345    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f0211-196d-4bc5-8725-a9a6d8bae6d1-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:46.431383 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.431378    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbprf\" (UniqueName: \"kubernetes.io/projected/717f0211-196d-4bc5-8725-a9a6d8bae6d1-kube-api-access-kbprf\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:46.495118 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.495081    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"]
Nov 02 02:16:46.498908 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.498881    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c942d-5fb8955c9-9k8fq"]
Nov 02 02:16:46.750503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.750462    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"]
Nov 02 02:16:46.755593 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:46.755561    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c942d-57ffddc647-dcr9n"]
Nov 02 02:16:47.225904 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:47.225873    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717f0211-196d-4bc5-8725-a9a6d8bae6d1" path="/var/lib/kubelet/pods/717f0211-196d-4bc5-8725-a9a6d8bae6d1/volumes"
Nov 02 02:16:47.226483 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:47.226464    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" path="/var/lib/kubelet/pods/eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6/volumes"
Nov 02 02:16:47.391809 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:47.391770    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:47.572447 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:47.572357    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:47.829830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:47.829757    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:16:47.836719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:47.836697    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:16:47.967759 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:47.967735    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:16:47.975099 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:47.975079    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:16:48.052408 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:48.052366    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:52.441769 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:52.441729    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:52.616798 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:52.616726    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:52.853694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:52.853600    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:53.373175 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:53.373146    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:16:57.651038 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:16:57.650991    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:01.662526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:01.662490    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"]
Nov 02 02:17:01.662953 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:01.662706    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl" podUID="cc459d6e-0d38-4103-959c-d11466ca4d0d" containerName="maas-api" containerID="cri-o://b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b" gracePeriod=30
Nov 02 02:17:01.798270 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:01.798237    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:01.947899 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:01.947868    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:17:02.002435 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.002407    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-metrics-certs\") pod \"cc459d6e-0d38-4103-959c-d11466ca4d0d\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") "
Nov 02 02:17:02.002651 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.002468    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc459d6e-0d38-4103-959c-d11466ca4d0d-kube-api-access-qplsk\" (UniqueName: \"kubernetes.io/projected/cc459d6e-0d38-4103-959c-d11466ca4d0d-kube-api-access-qplsk\") pod \"cc459d6e-0d38-4103-959c-d11466ca4d0d\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") "
Nov 02 02:17:02.002651 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.002640    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc459d6e-0d38-4103-959c-d11466ca4d0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc459d6e-0d38-4103-959c-d11466ca4d0d-openshift-service-ca\") pod \"cc459d6e-0d38-4103-959c-d11466ca4d0d\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") "
Nov 02 02:17:02.002793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.002672    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls\") pod \"cc459d6e-0d38-4103-959c-d11466ca4d0d\" (UID: \"cc459d6e-0d38-4103-959c-d11466ca4d0d\") "
Nov 02 02:17:02.003571 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.003515    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc459d6e-0d38-4103-959c-d11466ca4d0d-openshift-service-ca" pod "cc459d6e-0d38-4103-959c-d11466ca4d0d" (UID: "cc459d6e-0d38-4103-959c-d11466ca4d0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:02.004878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.004845    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-metrics-certs" pod "cc459d6e-0d38-4103-959c-d11466ca4d0d" (UID: "cc459d6e-0d38-4103-959c-d11466ca4d0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:02.005007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.004984    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc459d6e-0d38-4103-959c-d11466ca4d0d-kube-api-access-qplsk" pod "cc459d6e-0d38-4103-959c-d11466ca4d0d" (UID: "cc459d6e-0d38-4103-959c-d11466ca4d0d"). InnerVolumeSpecName "kube-api-access-qplsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:02.005106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.005016    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls" pod "cc459d6e-0d38-4103-959c-d11466ca4d0d" (UID: "cc459d6e-0d38-4103-959c-d11466ca4d0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:02.103976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.103937    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:02.104163 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.103980    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qplsk\" (UniqueName: \"kubernetes.io/projected/cc459d6e-0d38-4103-959c-d11466ca4d0d-kube-api-access-qplsk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:02.104163 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.103997    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc459d6e-0d38-4103-959c-d11466ca4d0d-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:02.104163 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.104012    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc459d6e-0d38-4103-959c-d11466ca4d0d-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:02.453931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.453891    4912 generic.go:356] "Generic (PLEG): container finished" podID="cc459d6e-0d38-4103-959c-d11466ca4d0d" containerID="b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b" exitCode=0
Nov 02 02:17:02.453931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.453934    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"
Nov 02 02:17:02.454227 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.453971    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl" event={"ID":"cc459d6e-0d38-4103-959c-d11466ca4d0d","Type":"ContainerDied","Data":"b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b"}
Nov 02 02:17:02.454227 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.454026    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl" event={"ID":"cc459d6e-0d38-4103-959c-d11466ca4d0d","Type":"ContainerDied","Data":"489859be08217a175fd7c9f1fe23a3d1930104cb5a76978884160c3929079129"}
Nov 02 02:17:02.454227 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.454048    4912 scope.go:122] "RemoveContainer" containerID="b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b"
Nov 02 02:17:02.466883 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.466861    4912 scope.go:122] "RemoveContainer" containerID="b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b"
Nov 02 02:17:02.467205 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:17:02.467184    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b\": container with ID starting with b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b not found: ID does not exist" containerID="b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b"
Nov 02 02:17:02.467312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.467215    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b"} err="failed to get container status \"b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b\": rpc error: code = NotFound desc = could not find container \"b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b\": container with ID starting with b6c5e22bd7e96f518c4a9d96a8ffbf8a05fdb34bec32ca7c821d5817ff438f3b not found: ID does not exist"
Nov 02 02:17:02.623865 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.623832    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"]
Nov 02 02:17:02.627621 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:02.627594    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4211c0-5f9d56644d-pppcl"]
Nov 02 02:17:03.225117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:03.225082    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc459d6e-0d38-4103-959c-d11466ca4d0d" path="/var/lib/kubelet/pods/cc459d6e-0d38-4103-959c-d11466ca4d0d/volumes"
Nov 02 02:17:03.317022 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:03.316989    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:05.793263 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:05.793221    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:08.628737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:08.628705    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:10.400979 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:10.400935    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:10.581768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:10.581732    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:10.765866 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:10.765765    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:11.234741 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:11.234699    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:12.138569 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:17:12.138532    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:17:13.554090 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:13.554024    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:16.167135 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:16.167099    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"]
Nov 02 02:17:16.216468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:16.216432    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"]
Nov 02 02:17:16.219414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:16.219386    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"]
Nov 02 02:17:16.285795 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:16.285763    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:18.649232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:18.649191    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:20.695308 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:20.695272    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:23.279776 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:23.279746    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"]
Nov 02 02:17:23.429350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:23.429320    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:23.659793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:23.659756    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:23.836415 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:23.836367    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:27.732508 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.732421    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"]
Nov 02 02:17:27.733823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733796    4912 cpu_manager.go:414] "containerMap: removing container" podUID="717f0211-196d-4bc5-8725-a9a6d8bae6d1" containerName="payload-pre-processing"
Nov 02 02:17:27.733823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733824    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717f0211-196d-4bc5-8725-a9a6d8bae6d1" containerName="payload-pre-processing"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733856    4912 cpu_manager.go:414] "containerMap: removing container" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerName="cleanup"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733865    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerName="cleanup"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733876    4912 cpu_manager.go:414] "containerMap: removing container" podUID="5f4daede-23e0-4199-834d-1a0ecf9cb72c" containerName="revoke-keys"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733884    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4daede-23e0-4199-834d-1a0ecf9cb72c" containerName="revoke-keys"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733898    4912 cpu_manager.go:414] "containerMap: removing container" podUID="eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" containerName="payload-processing"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733906    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" containerName="payload-processing"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733920    4912 cpu_manager.go:414] "containerMap: removing container" podUID="1a4bb112-0081-4f0f-9f12-914574e2a13a" containerName="main"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733928    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4bb112-0081-4f0f-9f12-914574e2a13a" containerName="main"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733947    4912 cpu_manager.go:414] "containerMap: removing container" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerName="cleanup"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733956    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerName="cleanup"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733966    4912 cpu_manager.go:414] "containerMap: removing container" podUID="cc459d6e-0d38-4103-959c-d11466ca4d0d" containerName="maas-api"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733971    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc459d6e-0d38-4103-959c-d11466ca4d0d" containerName="maas-api"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733983    4912 cpu_manager.go:414] "containerMap: removing container" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerName="cleanup"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733988    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerName="cleanup"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.733998    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerName="cleanup"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734002    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerName="cleanup"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734017    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy"
Nov 02 02:17:27.734017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734023    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734045    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerName="cleanup"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734073    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerName="cleanup"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734174    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="717f0211-196d-4bc5-8725-a9a6d8bae6d1" containerName="payload-pre-processing"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734191    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4daede-23e0-4199-834d-1a0ecf9cb72c" containerName="revoke-keys"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734200    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerName="cleanup"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734210    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafd3ce7-9463-4588-9fa5-2c9e43f6f5b6" containerName="payload-processing"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734219    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4bb112-0081-4f0f-9f12-914574e2a13a" containerName="main"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734228    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerName="cleanup"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734238    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc459d6e-0d38-4103-959c-d11466ca4d0d" containerName="maas-api"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734247    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c8faa9-9fc5-402e-b39b-8dc2c86c2b4f" containerName="cleanup"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734257    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerName="cleanup"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734267    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4633de5-ffcd-4056-8677-cd5426f8a643" containerName="istio-proxy"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734276    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerName="cleanup"
Nov 02 02:17:27.734830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.734285    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerName="cleanup"
Nov 02 02:17:27.737883 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.737864    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:27.788825 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.788795    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fztk\" (UniqueName: \"kubernetes.io/projected/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-kube-api-access-6fztk\") pod \"maas-api-revoke-sub-72aa13a4-2f5l6\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:27.788971 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.788834    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72aa13a4-2f5l6\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:27.789037 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.788957    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72aa13a4-2f5l6\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:27.794183 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.794155    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"]
Nov 02 02:17:27.839090 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.839028    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:17:27.839090 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.839028    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:17:27.890038 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.889992    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fztk\" (UniqueName: \"kubernetes.io/projected/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-kube-api-access-6fztk\") pod \"maas-api-revoke-sub-72aa13a4-2f5l6\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:27.890038 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.890043    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72aa13a4-2f5l6\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:27.890306 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.890128    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72aa13a4-2f5l6\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:27.890823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.890802    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72aa13a4-2f5l6\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:27.892741 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.892714    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72aa13a4-2f5l6\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:27.950941 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:27.950906    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fztk\" (UniqueName: \"kubernetes.io/projected/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-kube-api-access-6fztk\") pod \"maas-api-revoke-sub-72aa13a4-2f5l6\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:28.052336 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:28.052251    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:28.280117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:28.280079    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c" podUID="e9e74e12-b3db-4d35-9228-2a03fd7315d4" containerName="payload-pre-processing" containerID="cri-o://c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146" gracePeriod=25
Nov 02 02:17:28.433822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:28.433792    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"]
Nov 02 02:17:28.435713 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:17:28.435678    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b847a4d_9f5a_4e28_9972_d2bbc5dc01b1.slice/crio-d8577587f17ddbc860cb896e0c07cf2cd10349fc2d0911e3c5d75ec43b8343b7 WatchSource:0}: Error finding container d8577587f17ddbc860cb896e0c07cf2cd10349fc2d0911e3c5d75ec43b8343b7: Status 404 returned error can't find the container with id d8577587f17ddbc860cb896e0c07cf2cd10349fc2d0911e3c5d75ec43b8343b7
Nov 02 02:17:28.630147 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:28.630114    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6" event={"ID":"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1","Type":"ContainerStarted","Data":"d8577587f17ddbc860cb896e0c07cf2cd10349fc2d0911e3c5d75ec43b8343b7"}
Nov 02 02:17:29.637905 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:29.637871    4912 generic.go:356] "Generic (PLEG): container finished" podID="6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1" containerID="bd0317818548c10263a8d8bfb76ad5d7aee0621f09486849e67440a95421707b" exitCode=0
Nov 02 02:17:29.638362 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:29.637954    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6" event={"ID":"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1","Type":"ContainerDied","Data":"bd0317818548c10263a8d8bfb76ad5d7aee0621f09486849e67440a95421707b"}
Nov 02 02:17:30.797520 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:30.797495    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:30.924218 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:30.924136    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-kube-api-access-6fztk\" (UniqueName: \"kubernetes.io/projected/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-kube-api-access-6fztk\") pod \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") "
Nov 02 02:17:30.924397 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:30.924327    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-xks-ca\") pod \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") "
Nov 02 02:17:30.924397 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:30.924364    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-openshift-ca\") pod \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\" (UID: \"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1\") "
Nov 02 02:17:30.924774 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:30.924749    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-openshift-ca" pod "6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1" (UID: "6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:30.926627 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:30.926602    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-xks-ca" pod "6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1" (UID: "6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:30.926985 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:30.926966    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-kube-api-access-6fztk" pod "6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1" (UID: "6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1"). InnerVolumeSpecName "kube-api-access-6fztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:31.025830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.025794    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fztk\" (UniqueName: \"kubernetes.io/projected/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-kube-api-access-6fztk\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.025830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.025825    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.025830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.025836    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.189082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.188956    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main" containerID="cri-o://b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f" gracePeriod=45
Nov 02 02:17:31.237795 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.237759    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main" containerID="cri-o://1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1" gracePeriod=45
Nov 02 02:17:31.239322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.239290    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" containerID="cri-o://2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e" gracePeriod=45
Nov 02 02:17:31.252836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.252796    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": dial tcp 10.132.0.80:8000: connect: connection refused"
Nov 02 02:17:31.531302 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.531141    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 02:17:31.579203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.579175    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 02:17:31.588392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.587907    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 02:17:31.634825 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.634783    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-tmp-dir\") pod \"aefb2f44-77e8-45cb-9ec6-017183892791\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") "
Nov 02 02:17:31.635032 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.634862    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aefb2f44-77e8-45cb-9ec6-017183892791-tls-certs\" (UniqueName: \"kubernetes.io/secret/aefb2f44-77e8-45cb-9ec6-017183892791-tls-certs\") pod \"aefb2f44-77e8-45cb-9ec6-017183892791\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") "
Nov 02 02:17:31.635032 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.634891    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b841422-7581-414d-8ace-db272949f59d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b841422-7581-414d-8ace-db272949f59d-tls-certs\") pod \"4b841422-7581-414d-8ace-db272949f59d\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") "
Nov 02 02:17:31.635032 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.634944    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-model-cache\") pod \"aefb2f44-77e8-45cb-9ec6-017183892791\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") "
Nov 02 02:17:31.635032 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.634970    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-dshm\") pod \"4b841422-7581-414d-8ace-db272949f59d\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") "
Nov 02 02:17:31.635032 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635003    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-dshm\") pod \"aefb2f44-77e8-45cb-9ec6-017183892791\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") "
Nov 02 02:17:31.635423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635047    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-model-cache\") pod \"4b841422-7581-414d-8ace-db272949f59d\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") "
Nov 02 02:17:31.635423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635091    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-home\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-home\") pod \"aefb2f44-77e8-45cb-9ec6-017183892791\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") "
Nov 02 02:17:31.635423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635132    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e41a89-3a5d-4bb8-993c-8db65b10e224-kube-api-access-rmlqt\" (UniqueName: \"kubernetes.io/projected/24e41a89-3a5d-4bb8-993c-8db65b10e224-kube-api-access-rmlqt\") pod \"24e41a89-3a5d-4bb8-993c-8db65b10e224\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") "
Nov 02 02:17:31.635423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635172    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefb2f44-77e8-45cb-9ec6-017183892791-kube-api-access-mppxm\" (UniqueName: \"kubernetes.io/projected/aefb2f44-77e8-45cb-9ec6-017183892791-kube-api-access-mppxm\") pod \"aefb2f44-77e8-45cb-9ec6-017183892791\" (UID: \"aefb2f44-77e8-45cb-9ec6-017183892791\") "
Nov 02 02:17:31.635423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635238    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-dshm\") pod \"24e41a89-3a5d-4bb8-993c-8db65b10e224\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") "
Nov 02 02:17:31.635423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635270    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-tmp-dir\") pod \"24e41a89-3a5d-4bb8-993c-8db65b10e224\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") "
Nov 02 02:17:31.635423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635306    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-model-cache\") pod \"24e41a89-3a5d-4bb8-993c-8db65b10e224\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") "
Nov 02 02:17:31.635423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635336    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-home\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-home\") pod \"4b841422-7581-414d-8ace-db272949f59d\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") "
Nov 02 02:17:31.635423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635330    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-tmp-dir" pod "aefb2f44-77e8-45cb-9ec6-017183892791" (UID: "aefb2f44-77e8-45cb-9ec6-017183892791"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.635950 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635688    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-model-cache" pod "aefb2f44-77e8-45cb-9ec6-017183892791" (UID: "aefb2f44-77e8-45cb-9ec6-017183892791"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.635950 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635715    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-model-cache" pod "24e41a89-3a5d-4bb8-993c-8db65b10e224" (UID: "24e41a89-3a5d-4bb8-993c-8db65b10e224"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.635950 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.635933    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-tmp-dir" pod "24e41a89-3a5d-4bb8-993c-8db65b10e224" (UID: "24e41a89-3a5d-4bb8-993c-8db65b10e224"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.636204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.636181    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-home" pod "aefb2f44-77e8-45cb-9ec6-017183892791" (UID: "aefb2f44-77e8-45cb-9ec6-017183892791"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.636377 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.636358    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-model-cache" pod "4b841422-7581-414d-8ace-db272949f59d" (UID: "4b841422-7581-414d-8ace-db272949f59d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.636595 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.636573    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-home" pod "4b841422-7581-414d-8ace-db272949f59d" (UID: "4b841422-7581-414d-8ace-db272949f59d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.636822 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.636800    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b841422-7581-414d-8ace-db272949f59d-kube-api-access-zknbh\" (UniqueName: \"kubernetes.io/projected/4b841422-7581-414d-8ace-db272949f59d-kube-api-access-zknbh\") pod \"4b841422-7581-414d-8ace-db272949f59d\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") "
Nov 02 02:17:31.636993 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.636975    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24e41a89-3a5d-4bb8-993c-8db65b10e224-tls-certs\" (UniqueName: \"kubernetes.io/secret/24e41a89-3a5d-4bb8-993c-8db65b10e224-tls-certs\") pod \"24e41a89-3a5d-4bb8-993c-8db65b10e224\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") "
Nov 02 02:17:31.637146 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.637129    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-tmp-dir\") pod \"4b841422-7581-414d-8ace-db272949f59d\" (UID: \"4b841422-7581-414d-8ace-db272949f59d\") "
Nov 02 02:17:31.637764 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.637743    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-home\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-home\") pod \"24e41a89-3a5d-4bb8-993c-8db65b10e224\" (UID: \"24e41a89-3a5d-4bb8-993c-8db65b10e224\") "
Nov 02 02:17:31.638476 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.638451    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.638616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.638601    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.638712 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.638700    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.638798 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.638788    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.638882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.638871    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.639004 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.638990    4912 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-model-cache\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.639129 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.639116    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.639219 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.638475    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-dshm" pod "24e41a89-3a5d-4bb8-993c-8db65b10e224" (UID: "24e41a89-3a5d-4bb8-993c-8db65b10e224"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.639299 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.638612    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b841422-7581-414d-8ace-db272949f59d-tls-certs" pod "4b841422-7581-414d-8ace-db272949f59d" (UID: "4b841422-7581-414d-8ace-db272949f59d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:31.639382 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.638883    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-tmp-dir" pod "4b841422-7581-414d-8ace-db272949f59d" (UID: "4b841422-7581-414d-8ace-db272949f59d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.639495 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.639321    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-home" pod "24e41a89-3a5d-4bb8-993c-8db65b10e224" (UID: "24e41a89-3a5d-4bb8-993c-8db65b10e224"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.639631 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.639382    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aefb2f44-77e8-45cb-9ec6-017183892791-tls-certs" pod "aefb2f44-77e8-45cb-9ec6-017183892791" (UID: "aefb2f44-77e8-45cb-9ec6-017183892791"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:31.640088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.640045    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-dshm" pod "4b841422-7581-414d-8ace-db272949f59d" (UID: "4b841422-7581-414d-8ace-db272949f59d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.640197 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.640087    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefb2f44-77e8-45cb-9ec6-017183892791-kube-api-access-mppxm" pod "aefb2f44-77e8-45cb-9ec6-017183892791" (UID: "aefb2f44-77e8-45cb-9ec6-017183892791"). InnerVolumeSpecName "kube-api-access-mppxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:31.640197 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.640126    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e41a89-3a5d-4bb8-993c-8db65b10e224-kube-api-access-rmlqt" pod "24e41a89-3a5d-4bb8-993c-8db65b10e224" (UID: "24e41a89-3a5d-4bb8-993c-8db65b10e224"). InnerVolumeSpecName "kube-api-access-rmlqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:31.640694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.640674    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e41a89-3a5d-4bb8-993c-8db65b10e224-tls-certs" pod "24e41a89-3a5d-4bb8-993c-8db65b10e224" (UID: "24e41a89-3a5d-4bb8-993c-8db65b10e224"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:31.640878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.640857    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-dshm" pod "aefb2f44-77e8-45cb-9ec6-017183892791" (UID: "aefb2f44-77e8-45cb-9ec6-017183892791"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:31.642037 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.642014    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b841422-7581-414d-8ace-db272949f59d-kube-api-access-zknbh" pod "4b841422-7581-414d-8ace-db272949f59d" (UID: "4b841422-7581-414d-8ace-db272949f59d"). InnerVolumeSpecName "kube-api-access-zknbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:31.656769 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.656730    4912 generic.go:356] "Generic (PLEG): container finished" podID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerID="2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e" exitCode=0
Nov 02 02:17:31.657008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.656787    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"
Nov 02 02:17:31.657008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.656866    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" event={"ID":"24e41a89-3a5d-4bb8-993c-8db65b10e224","Type":"ContainerDied","Data":"2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e"}
Nov 02 02:17:31.658539 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.658507    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf" event={"ID":"24e41a89-3a5d-4bb8-993c-8db65b10e224","Type":"ContainerDied","Data":"0012ed5fc6a8e526f391e779825b735a725848de4e5ea2dd17d298ad664a896b"}
Nov 02 02:17:31.658693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.658533    4912 scope.go:122] "RemoveContainer" containerID="2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e"
Nov 02 02:17:31.664940 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.664912    4912 generic.go:356] "Generic (PLEG): container finished" podID="aefb2f44-77e8-45cb-9ec6-017183892791" containerID="1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1" exitCode=0
Nov 02 02:17:31.665396 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.665201    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"
Nov 02 02:17:31.665726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.665219    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" event={"ID":"aefb2f44-77e8-45cb-9ec6-017183892791","Type":"ContainerDied","Data":"1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1"}
Nov 02 02:17:31.665910 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.665883    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622" event={"ID":"aefb2f44-77e8-45cb-9ec6-017183892791","Type":"ContainerDied","Data":"f2e0ff17ebab4e9f181152778a226961922877da3764dbee59e6ecc13b4acda2"}
Nov 02 02:17:31.667591 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.667564    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6" event={"ID":"6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1","Type":"ContainerDied","Data":"d8577587f17ddbc860cb896e0c07cf2cd10349fc2d0911e3c5d75ec43b8343b7"}
Nov 02 02:17:31.667724 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.667649    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8577587f17ddbc860cb896e0c07cf2cd10349fc2d0911e3c5d75ec43b8343b7"
Nov 02 02:17:31.667724 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.667719    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"
Nov 02 02:17:31.672138 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.672117    4912 generic.go:356] "Generic (PLEG): container finished" podID="4b841422-7581-414d-8ace-db272949f59d" containerID="b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f" exitCode=0
Nov 02 02:17:31.672262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.672197    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" event={"ID":"4b841422-7581-414d-8ace-db272949f59d","Type":"ContainerDied","Data":"b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f"}
Nov 02 02:17:31.672262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.672217    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn" event={"ID":"4b841422-7581-414d-8ace-db272949f59d","Type":"ContainerDied","Data":"593d3c08e21a1f8c5d740b8582dba0b0d7d2750193e252b7bb2e208f3eabcd59"}
Nov 02 02:17:31.672262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.672262    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"
Nov 02 02:17:31.686261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.686238    4912 scope.go:122] "RemoveContainer" containerID="2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e"
Nov 02 02:17:31.686573 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:17:31.686553    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e\": container with ID starting with 2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e not found: ID does not exist" containerID="2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e"
Nov 02 02:17:31.686654 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.686584    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e"} err="failed to get container status \"2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e\": rpc error: code = NotFound desc = could not find container \"2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e\": container with ID starting with 2f5495d4cf7c248732e4d8e75677fcb0385d4c1e2b7bf209c6595c914da2960e not found: ID does not exist"
Nov 02 02:17:31.686654 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.686601    4912 scope.go:122] "RemoveContainer" containerID="1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1"
Nov 02 02:17:31.709334 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.700797    4912 scope.go:122] "RemoveContainer" containerID="1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1"
Nov 02 02:17:31.709334 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:17:31.701230    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1\": container with ID starting with 1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1 not found: ID does not exist" containerID="1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1"
Nov 02 02:17:31.709334 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.701269    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1"} err="failed to get container status \"1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1\": rpc error: code = NotFound desc = could not find container \"1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1\": container with ID starting with 1ce42e0e2bb57822e6796c33d06b1cfcc0dfd0210153e82e83f5581e1ec29ab1 not found: ID does not exist"
Nov 02 02:17:31.709334 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.701297    4912 scope.go:122] "RemoveContainer" containerID="b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f"
Nov 02 02:17:31.714814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.714785    4912 scope.go:122] "RemoveContainer" containerID="b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f"
Nov 02 02:17:31.715195 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:17:31.715168    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f\": container with ID starting with b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f not found: ID does not exist" containerID="b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f"
Nov 02 02:17:31.715297 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.715208    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f"} err="failed to get container status \"b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f\": rpc error: code = NotFound desc = could not find container \"b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f\": container with ID starting with b85f3ea954da0233f33739e67d8378a74818d69d7ff6bb9d6fdebbab3ef8718f not found: ID does not exist"
Nov 02 02:17:31.740106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740012    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mppxm\" (UniqueName: \"kubernetes.io/projected/aefb2f44-77e8-45cb-9ec6-017183892791-kube-api-access-mppxm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740044    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740071    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zknbh\" (UniqueName: \"kubernetes.io/projected/4b841422-7581-414d-8ace-db272949f59d-kube-api-access-zknbh\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740085    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24e41a89-3a5d-4bb8-993c-8db65b10e224-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740098    4912 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-tmp-dir\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740107    4912 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24e41a89-3a5d-4bb8-993c-8db65b10e224-home\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740115    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefb2f44-77e8-45cb-9ec6-017183892791-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740123    4912 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b841422-7581-414d-8ace-db272949f59d-tls-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740131    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b841422-7581-414d-8ace-db272949f59d-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740138    4912 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aefb2f44-77e8-45cb-9ec6-017183892791-dshm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:31.740429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:31.740147    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmlqt\" (UniqueName: \"kubernetes.io/projected/24e41a89-3a5d-4bb8-993c-8db65b10e224-kube-api-access-rmlqt\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:32.010329 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:32.010246    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"]
Nov 02 02:17:32.013288 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:32.013258    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/distinct-w6-0ebdb3-kserve-89ddf8cd9-xw9vn"]
Nov 02 02:17:32.171203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:32.171162    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"]
Nov 02 02:17:32.175794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:32.175767    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/simulator-w6-0ebdb3-kserve-985cc5f47-j9slf"]
Nov 02 02:17:32.283963 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:32.283878    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"]
Nov 02 02:17:32.290603 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:32.290574    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0ebdb3/premium-w6-0ebdb3-kserve-784d7588fd-6l622"]
Nov 02 02:17:33.224198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:33.224164    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" path="/var/lib/kubelet/pods/24e41a89-3a5d-4bb8-993c-8db65b10e224/volumes"
Nov 02 02:17:33.224744 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:33.224630    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b841422-7581-414d-8ace-db272949f59d" path="/var/lib/kubelet/pods/4b841422-7581-414d-8ace-db272949f59d/volumes"
Nov 02 02:17:33.225027 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:33.225011    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" path="/var/lib/kubelet/pods/aefb2f44-77e8-45cb-9ec6-017183892791/volumes"
Nov 02 02:17:37.743423 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:37.743386    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c" podUID="e9e74e12-b3db-4d35-9228-2a03fd7315d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:17:42.138575 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:17:42.138537    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:17:43.199616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:43.199577    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:44.649705 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:44.649667    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:44.932864 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:44.932784    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:45.652672 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:45.652616    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:47.743082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:47.743022    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c" podUID="e9e74e12-b3db-4d35-9228-2a03fd7315d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:17:48.537903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.537844    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 02:17:48.725870 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.725832    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e74e12-b3db-4d35-9228-2a03fd7315d4-kube-api-access-ljh5b\" (UniqueName: \"kubernetes.io/projected/e9e74e12-b3db-4d35-9228-2a03fd7315d4-kube-api-access-ljh5b\") pod \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\" (UID: \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\") "
Nov 02 02:17:48.726077 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.725890    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e74e12-b3db-4d35-9228-2a03fd7315d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e74e12-b3db-4d35-9228-2a03fd7315d4-plugins-config-volume\") pod \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\" (UID: \"e9e74e12-b3db-4d35-9228-2a03fd7315d4\") "
Nov 02 02:17:48.726395 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.726367    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e74e12-b3db-4d35-9228-2a03fd7315d4-plugins-config-volume" pod "e9e74e12-b3db-4d35-9228-2a03fd7315d4" (UID: "e9e74e12-b3db-4d35-9228-2a03fd7315d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:48.728376 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.728344    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e74e12-b3db-4d35-9228-2a03fd7315d4-kube-api-access-ljh5b" pod "e9e74e12-b3db-4d35-9228-2a03fd7315d4" (UID: "e9e74e12-b3db-4d35-9228-2a03fd7315d4"). InnerVolumeSpecName "kube-api-access-ljh5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:48.791277 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.791178    4912 generic.go:356] "Generic (PLEG): container finished" podID="e9e74e12-b3db-4d35-9228-2a03fd7315d4" containerID="c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146" exitCode=0
Nov 02 02:17:48.791277 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.791244    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"
Nov 02 02:17:48.791277 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.791262    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c" event={"ID":"e9e74e12-b3db-4d35-9228-2a03fd7315d4","Type":"ContainerDied","Data":"c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146"}
Nov 02 02:17:48.791940 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.791351    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c" event={"ID":"e9e74e12-b3db-4d35-9228-2a03fd7315d4","Type":"ContainerDied","Data":"f55082dba3103e9f7c4898b043f3b73f7bcae8ef19bc4566622e83fcb1eec612"}
Nov 02 02:17:48.791940 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.791381    4912 scope.go:122] "RemoveContainer" containerID="c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146"
Nov 02 02:17:48.803402 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.803380    4912 scope.go:122] "RemoveContainer" containerID="c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146"
Nov 02 02:17:48.803693 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:17:48.803670    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146\": container with ID starting with c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146 not found: ID does not exist" containerID="c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146"
Nov 02 02:17:48.803796 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.803701    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146"} err="failed to get container status \"c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146\": rpc error: code = NotFound desc = could not find container \"c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146\": container with ID starting with c29c98d8089bd07479b93d651077f06f8451c72152d1ed8d77b9ea074222e146 not found: ID does not exist"
Nov 02 02:17:48.827503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.827461    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljh5b\" (UniqueName: \"kubernetes.io/projected/e9e74e12-b3db-4d35-9228-2a03fd7315d4-kube-api-access-ljh5b\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:48.827503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.827502    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e74e12-b3db-4d35-9228-2a03fd7315d4-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:48.956916 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.956882    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"]
Nov 02 02:17:48.963213 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:48.963182    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ebdb3-859b4f5b67-8dz4c"]
Nov 02 02:17:49.224910 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:49.224878    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e74e12-b3db-4d35-9228-2a03fd7315d4" path="/var/lib/kubelet/pods/e9e74e12-b3db-4d35-9228-2a03fd7315d4/volumes"
Nov 02 02:17:52.435761 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:52.435718    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:52.715892 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:52.715812    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:52.782160 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:52.782126    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"]
Nov 02 02:17:52.782354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:52.782311    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy" containerID="cri-o://8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9" gracePeriod=30
Nov 02 02:17:52.909903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:52.909865    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:53.611096 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:53.611040    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:53.914536 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:53.914491    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:54.383835 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:54.383796    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:55.022878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:55.022831    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:55.202513 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:55.202475    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:55.380628 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:55.380592    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:17:58.041176 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.041151    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 02:17:58.131586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131553    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-istio-token\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-istio-token\") pod \"abe4c650-474e-4284-8c44-0c971f95efe5\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") "
Nov 02 02:17:58.131779 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131633    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-data\") pod \"abe4c650-474e-4284-8c44-0c971f95efe5\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") "
Nov 02 02:17:58.131779 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131668    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-certs\") pod \"abe4c650-474e-4284-8c44-0c971f95efe5\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") "
Nov 02 02:17:58.131779 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131713    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe4c650-474e-4284-8c44-0c971f95efe5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe4c650-474e-4284-8c44-0c971f95efe5-istiod-ca-cert\") pod \"abe4c650-474e-4284-8c44-0c971f95efe5\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") "
Nov 02 02:17:58.131779 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131769    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-kube-api-access-2xt5k\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-kube-api-access-2xt5k\") pod \"abe4c650-474e-4284-8c44-0c971f95efe5\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") "
Nov 02 02:17:58.132021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131805    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-envoy\") pod \"abe4c650-474e-4284-8c44-0c971f95efe5\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") "
Nov 02 02:17:58.132021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131868    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-credential-socket\") pod \"abe4c650-474e-4284-8c44-0c971f95efe5\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") "
Nov 02 02:17:58.132021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131916    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-socket\") pod \"abe4c650-474e-4284-8c44-0c971f95efe5\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") "
Nov 02 02:17:58.132021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131995    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abe4c650-474e-4284-8c44-0c971f95efe5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe4c650-474e-4284-8c44-0c971f95efe5-istio-podinfo\") pod \"abe4c650-474e-4284-8c44-0c971f95efe5\" (UID: \"abe4c650-474e-4284-8c44-0c971f95efe5\") "
Nov 02 02:17:58.132021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.131978    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-certs" pod "abe4c650-474e-4284-8c44-0c971f95efe5" (UID: "abe4c650-474e-4284-8c44-0c971f95efe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:58.132021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.132012    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-data" pod "abe4c650-474e-4284-8c44-0c971f95efe5" (UID: "abe4c650-474e-4284-8c44-0c971f95efe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:58.132387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.132234    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-credential-socket" pod "abe4c650-474e-4284-8c44-0c971f95efe5" (UID: "abe4c650-474e-4284-8c44-0c971f95efe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:58.132387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.132251    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe4c650-474e-4284-8c44-0c971f95efe5-istiod-ca-cert" pod "abe4c650-474e-4284-8c44-0c971f95efe5" (UID: "abe4c650-474e-4284-8c44-0c971f95efe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:58.132387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.132262    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-socket" pod "abe4c650-474e-4284-8c44-0c971f95efe5" (UID: "abe4c650-474e-4284-8c44-0c971f95efe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:58.132387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.132346    4912 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.132387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.132365    4912 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-data\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.132387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.132380    4912 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-workload-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.132876 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.132392    4912 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe4c650-474e-4284-8c44-0c971f95efe5-istiod-ca-cert\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.132876 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.132406    4912 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-credential-socket\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.134306 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.134269    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-istio-token" pod "abe4c650-474e-4284-8c44-0c971f95efe5" (UID: "abe4c650-474e-4284-8c44-0c971f95efe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:58.134406 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.134374    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-envoy" pod "abe4c650-474e-4284-8c44-0c971f95efe5" (UID: "abe4c650-474e-4284-8c44-0c971f95efe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:58.134797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.134774    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abe4c650-474e-4284-8c44-0c971f95efe5-istio-podinfo" pod "abe4c650-474e-4284-8c44-0c971f95efe5" (UID: "abe4c650-474e-4284-8c44-0c971f95efe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:17:58.134867 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.134787    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-kube-api-access-2xt5k" pod "abe4c650-474e-4284-8c44-0c971f95efe5" (UID: "abe4c650-474e-4284-8c44-0c971f95efe5"). InnerVolumeSpecName "kube-api-access-2xt5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:58.233218 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.233184    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xt5k\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-kube-api-access-2xt5k\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.233218 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.233216    4912 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe4c650-474e-4284-8c44-0c971f95efe5-istio-envoy\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.233451 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.233228    4912 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe4c650-474e-4284-8c44-0c971f95efe5-istio-podinfo\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.233451 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.233239    4912 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe4c650-474e-4284-8c44-0c971f95efe5-istio-token\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.856777 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.856736    4912 generic.go:356] "Generic (PLEG): container finished" podID="abe4c650-474e-4284-8c44-0c971f95efe5" containerID="8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9" exitCode=0
Nov 02 02:17:58.856965 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.856797    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"
Nov 02 02:17:58.856965 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.856834    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" event={"ID":"abe4c650-474e-4284-8c44-0c971f95efe5","Type":"ContainerDied","Data":"8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9"}
Nov 02 02:17:58.856965 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.856914    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx" event={"ID":"abe4c650-474e-4284-8c44-0c971f95efe5","Type":"ContainerDied","Data":"7ec53cc8606115cd1ed3461f112ba2d9d0847f0ae5f2ce3d9271edf4b626fd23"}
Nov 02 02:17:58.856965 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.856953    4912 scope.go:122] "RemoveContainer" containerID="8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9"
Nov 02 02:17:58.870213 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.870194    4912 scope.go:122] "RemoveContainer" containerID="8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9"
Nov 02 02:17:58.870520 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:17:58.870487    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9\": container with ID starting with 8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9 not found: ID does not exist" containerID="8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9"
Nov 02 02:17:58.870603 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:58.870522    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9"} err="failed to get container status \"8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9\": rpc error: code = NotFound desc = could not find container \"8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9\": container with ID starting with 8764490c5677ae30c5608063ff6e51c5065bcb6afdaba9a66625bf17a45b10c9 not found: ID does not exist"
Nov 02 02:17:59.074051 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:59.074019    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"]
Nov 02 02:17:59.123360 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:59.123333    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-8l2cx"]
Nov 02 02:17:59.224590 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:17:59.224558    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" path="/var/lib/kubelet/pods/abe4c650-474e-4284-8c44-0c971f95efe5/volumes"
Nov 02 02:18:01.135774 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:01.135733    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:18:12.137870 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:18:12.137830    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:18:21.912615 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.912575    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"]
Nov 02 02:18:21.913253 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913237    4912 cpu_manager.go:414] "containerMap: removing container" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main"
Nov 02 02:18:21.913311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913256    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main"
Nov 02 02:18:21.913311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913284    4912 cpu_manager.go:414] "containerMap: removing container" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerName="cleanup"
Nov 02 02:18:21.913311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913292    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd75739-b39a-423a-8ddd-9a01cf4a93cf" containerName="cleanup"
Nov 02 02:18:21.913410 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913310    4912 cpu_manager.go:414] "containerMap: removing container" podUID="e9e74e12-b3db-4d35-9228-2a03fd7315d4" containerName="payload-pre-processing"
Nov 02 02:18:21.913410 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913318    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e74e12-b3db-4d35-9228-2a03fd7315d4" containerName="payload-pre-processing"
Nov 02 02:18:21.913410 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913343    4912 cpu_manager.go:414] "containerMap: removing container" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy"
Nov 02 02:18:21.913410 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913351    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy"
Nov 02 02:18:21.913410 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913364    4912 cpu_manager.go:414] "containerMap: removing container" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main"
Nov 02 02:18:21.913410 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913372    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main"
Nov 02 02:18:21.913410 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913390    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1" containerName="revoke-keys"
Nov 02 02:18:21.913410 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913397    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1" containerName="revoke-keys"
Nov 02 02:18:21.913667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913415    4912 cpu_manager.go:414] "containerMap: removing container" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main"
Nov 02 02:18:21.913667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913422    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main"
Nov 02 02:18:21.913667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913521    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b841422-7581-414d-8ace-db272949f59d" containerName="main"
Nov 02 02:18:21.913667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913534    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e41a89-3a5d-4bb8-993c-8db65b10e224" containerName="main"
Nov 02 02:18:21.913667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913547    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e74e12-b3db-4d35-9228-2a03fd7315d4" containerName="payload-pre-processing"
Nov 02 02:18:21.913667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913559    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe4c650-474e-4284-8c44-0c971f95efe5" containerName="istio-proxy"
Nov 02 02:18:21.913667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913569    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="aefb2f44-77e8-45cb-9ec6-017183892791" containerName="main"
Nov 02 02:18:21.913667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.913579    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1" containerName="revoke-keys"
Nov 02 02:18:21.917308 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.917286    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:21.972799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:21.972761    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"]
Nov 02 02:18:22.010029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.009995    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvct\" (UniqueName: \"kubernetes.io/projected/79c39264-71a2-4166-b6e8-d6e0db7eb591-kube-api-access-qhvct\") pod \"maas-api-revoke-sub-85237e45-wrcqd\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.010029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.010031    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wrcqd\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.010287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.010233    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wrcqd\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.018507 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.018482    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:18:22.018593 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.018555    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:18:22.110980 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.110938    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wrcqd\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.111209 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.110996    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvct\" (UniqueName: \"kubernetes.io/projected/79c39264-71a2-4166-b6e8-d6e0db7eb591-kube-api-access-qhvct\") pod \"maas-api-revoke-sub-85237e45-wrcqd\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.111209 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.111018    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wrcqd\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.111787 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.111763    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wrcqd\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.113485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.113464    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wrcqd\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.171700 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.171633    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvct\" (UniqueName: \"kubernetes.io/projected/79c39264-71a2-4166-b6e8-d6e0db7eb591-kube-api-access-qhvct\") pod \"maas-api-revoke-sub-85237e45-wrcqd\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.231354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.231321    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:22.611520 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:22.611476    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"]
Nov 02 02:18:22.612917 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:18:22.612890    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c39264_71a2_4166_b6e8_d6e0db7eb591.slice/crio-ea5be68f7fa6e6542e076c305bfc6ad6f58da170a613dc13c4d165e7250f035d WatchSource:0}: Error finding container ea5be68f7fa6e6542e076c305bfc6ad6f58da170a613dc13c4d165e7250f035d: Status 404 returned error can't find the container with id ea5be68f7fa6e6542e076c305bfc6ad6f58da170a613dc13c4d165e7250f035d
Nov 02 02:18:23.043375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:23.043340    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd" event={"ID":"79c39264-71a2-4166-b6e8-d6e0db7eb591","Type":"ContainerStarted","Data":"4dda39d3eb2d9d85cc1d71b2a954f690a39e67a06e77670dd8dd5631635a3ad8"}
Nov 02 02:18:23.043829 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:23.043397    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd" event={"ID":"79c39264-71a2-4166-b6e8-d6e0db7eb591","Type":"ContainerStarted","Data":"ea5be68f7fa6e6542e076c305bfc6ad6f58da170a613dc13c4d165e7250f035d"}
Nov 02 02:18:23.158919 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:23.158865    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd" podStartSLOduration=2.158849946 podStartE2EDuration="2.158849946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:18:23.157626547 +0000 UTC m=+2796.063091904" watchObservedRunningTime="2026-11-02 02:18:23.158849946 +0000 UTC m=+2796.064315331"
Nov 02 02:18:24.051805 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:24.051766    4912 generic.go:356] "Generic (PLEG): container finished" podID="79c39264-71a2-4166-b6e8-d6e0db7eb591" containerID="4dda39d3eb2d9d85cc1d71b2a954f690a39e67a06e77670dd8dd5631635a3ad8" exitCode=0
Nov 02 02:18:24.052302 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:24.051838    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd" event={"ID":"79c39264-71a2-4166-b6e8-d6e0db7eb591","Type":"ContainerDied","Data":"4dda39d3eb2d9d85cc1d71b2a954f690a39e67a06e77670dd8dd5631635a3ad8"}
Nov 02 02:18:25.196127 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.196102    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:25.243136 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.243110    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-openshift-ca\") pod \"79c39264-71a2-4166-b6e8-d6e0db7eb591\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") "
Nov 02 02:18:25.243306 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.243145    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-xks-ca\") pod \"79c39264-71a2-4166-b6e8-d6e0db7eb591\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") "
Nov 02 02:18:25.243306 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.243235    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c39264-71a2-4166-b6e8-d6e0db7eb591-kube-api-access-qhvct\" (UniqueName: \"kubernetes.io/projected/79c39264-71a2-4166-b6e8-d6e0db7eb591-kube-api-access-qhvct\") pod \"79c39264-71a2-4166-b6e8-d6e0db7eb591\" (UID: \"79c39264-71a2-4166-b6e8-d6e0db7eb591\") "
Nov 02 02:18:25.243445 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.243418    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-openshift-ca" pod "79c39264-71a2-4166-b6e8-d6e0db7eb591" (UID: "79c39264-71a2-4166-b6e8-d6e0db7eb591"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:18:25.243528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.243507    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:25.245430 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.245395    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-xks-ca" pod "79c39264-71a2-4166-b6e8-d6e0db7eb591" (UID: "79c39264-71a2-4166-b6e8-d6e0db7eb591"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:18:25.245555 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.245536    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c39264-71a2-4166-b6e8-d6e0db7eb591-kube-api-access-qhvct" pod "79c39264-71a2-4166-b6e8-d6e0db7eb591" (UID: "79c39264-71a2-4166-b6e8-d6e0db7eb591"). InnerVolumeSpecName "kube-api-access-qhvct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:18:25.344538 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.344423    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c39264-71a2-4166-b6e8-d6e0db7eb591-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:25.344538 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:25.344460    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhvct\" (UniqueName: \"kubernetes.io/projected/79c39264-71a2-4166-b6e8-d6e0db7eb591-kube-api-access-qhvct\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:26.065521 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:26.065491    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd" event={"ID":"79c39264-71a2-4166-b6e8-d6e0db7eb591","Type":"ContainerDied","Data":"ea5be68f7fa6e6542e076c305bfc6ad6f58da170a613dc13c4d165e7250f035d"}
Nov 02 02:18:26.065713 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:26.065533    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"
Nov 02 02:18:26.065713 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:26.065540    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea5be68f7fa6e6542e076c305bfc6ad6f58da170a613dc13c4d165e7250f035d"
Nov 02 02:18:34.365386 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:34.365346    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:18:37.081878 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:37.081843    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:18:42.138013 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:18:42.137975    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:18:54.051514 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:18:54.051478    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:19:12.137805 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:19:12.137717    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:19:31.103636 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.103601    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"]
Nov 02 02:19:31.104134 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.104124    4912 cpu_manager.go:414] "containerMap: removing container" podUID="79c39264-71a2-4166-b6e8-d6e0db7eb591" containerName="revoke-keys"
Nov 02 02:19:31.104182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.104136    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c39264-71a2-4166-b6e8-d6e0db7eb591" containerName="revoke-keys"
Nov 02 02:19:31.104266 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.104254    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c39264-71a2-4166-b6e8-d6e0db7eb591" containerName="revoke-keys"
Nov 02 02:19:31.107519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.107502    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.160595 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.160563    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:19:31.164414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.164381    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"]
Nov 02 02:19:31.194606 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.194579    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vxr\" (UniqueName: \"kubernetes.io/projected/319b3f4a-962f-4eae-b2fa-3facce48ec17-kube-api-access-46vxr\") pod \"maas-api-revoke-sub-c1e271a9-jfdrj\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.194768 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.194636    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfdrj\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.194841 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.194775    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfdrj\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.209961 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.209919    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:19:31.296187 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.296144    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vxr\" (UniqueName: \"kubernetes.io/projected/319b3f4a-962f-4eae-b2fa-3facce48ec17-kube-api-access-46vxr\") pod \"maas-api-revoke-sub-c1e271a9-jfdrj\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.296386 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.296213    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfdrj\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.296386 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.296260    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfdrj\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.297023 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.296999    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfdrj\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.298789 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.298767    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfdrj\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.357390 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.357322    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vxr\" (UniqueName: \"kubernetes.io/projected/319b3f4a-962f-4eae-b2fa-3facce48ec17-kube-api-access-46vxr\") pod \"maas-api-revoke-sub-c1e271a9-jfdrj\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.420882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.420849    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:31.794090 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:31.794039    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"]
Nov 02 02:19:31.795284 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:19:31.795256    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319b3f4a_962f_4eae_b2fa_3facce48ec17.slice/crio-ba7c873fbb94555a22018af17e27aa8383c46e60983dc9f89bacfacb0fe95044 WatchSource:0}: Error finding container ba7c873fbb94555a22018af17e27aa8383c46e60983dc9f89bacfacb0fe95044: Status 404 returned error can't find the container with id ba7c873fbb94555a22018af17e27aa8383c46e60983dc9f89bacfacb0fe95044
Nov 02 02:19:32.492968 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:32.492933    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj" event={"ID":"319b3f4a-962f-4eae-b2fa-3facce48ec17","Type":"ContainerStarted","Data":"78a6b6ca947516700e81d23dbc98e294292bb5728d18a5ac569bccf38ccb8da1"}
Nov 02 02:19:32.493498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:32.492984    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj" event={"ID":"319b3f4a-962f-4eae-b2fa-3facce48ec17","Type":"ContainerStarted","Data":"ba7c873fbb94555a22018af17e27aa8383c46e60983dc9f89bacfacb0fe95044"}
Nov 02 02:19:32.571569 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:32.571528    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:19:32.609947 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:32.609897    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj" podStartSLOduration=1.609884689 podStartE2EDuration="1.609884689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:19:32.608871958 +0000 UTC m=+2865.514337316" watchObservedRunningTime="2026-11-02 02:19:32.609884689 +0000 UTC m=+2865.515350045"
Nov 02 02:19:32.751568 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:32.751487    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:19:33.500152 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:33.500119    4912 generic.go:356] "Generic (PLEG): container finished" podID="319b3f4a-962f-4eae-b2fa-3facce48ec17" containerID="78a6b6ca947516700e81d23dbc98e294292bb5728d18a5ac569bccf38ccb8da1" exitCode=0
Nov 02 02:19:33.500592 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:33.500205    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj" event={"ID":"319b3f4a-962f-4eae-b2fa-3facce48ec17","Type":"ContainerDied","Data":"78a6b6ca947516700e81d23dbc98e294292bb5728d18a5ac569bccf38ccb8da1"}
Nov 02 02:19:34.651715 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.651693    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:34.732166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.732126    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-xks-ca\") pod \"319b3f4a-962f-4eae-b2fa-3facce48ec17\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") "
Nov 02 02:19:34.732375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.732257    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-openshift-ca\") pod \"319b3f4a-962f-4eae-b2fa-3facce48ec17\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") "
Nov 02 02:19:34.732375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.732307    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319b3f4a-962f-4eae-b2fa-3facce48ec17-kube-api-access-46vxr\" (UniqueName: \"kubernetes.io/projected/319b3f4a-962f-4eae-b2fa-3facce48ec17-kube-api-access-46vxr\") pod \"319b3f4a-962f-4eae-b2fa-3facce48ec17\" (UID: \"319b3f4a-962f-4eae-b2fa-3facce48ec17\") "
Nov 02 02:19:34.732668 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.732647    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-openshift-ca" pod "319b3f4a-962f-4eae-b2fa-3facce48ec17" (UID: "319b3f4a-962f-4eae-b2fa-3facce48ec17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:19:34.734598 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.734566    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-xks-ca" pod "319b3f4a-962f-4eae-b2fa-3facce48ec17" (UID: "319b3f4a-962f-4eae-b2fa-3facce48ec17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:19:34.734749 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.734686    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319b3f4a-962f-4eae-b2fa-3facce48ec17-kube-api-access-46vxr" pod "319b3f4a-962f-4eae-b2fa-3facce48ec17" (UID: "319b3f4a-962f-4eae-b2fa-3facce48ec17"). InnerVolumeSpecName "kube-api-access-46vxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:19:34.834260 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.834173    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:19:34.834260 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.834202    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46vxr\" (UniqueName: \"kubernetes.io/projected/319b3f4a-962f-4eae-b2fa-3facce48ec17-kube-api-access-46vxr\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:19:34.834260 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:34.834214    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319b3f4a-962f-4eae-b2fa-3facce48ec17-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:19:35.514429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:35.514339    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"
Nov 02 02:19:35.514429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:35.514380    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj" event={"ID":"319b3f4a-962f-4eae-b2fa-3facce48ec17","Type":"ContainerDied","Data":"ba7c873fbb94555a22018af17e27aa8383c46e60983dc9f89bacfacb0fe95044"}
Nov 02 02:19:35.514632 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:35.514441    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba7c873fbb94555a22018af17e27aa8383c46e60983dc9f89bacfacb0fe95044"
Nov 02 02:19:42.137529 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:19:42.137495    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:19:54.405528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:19:54.405487    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:20:02.049455 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.049413    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"]
Nov 02 02:20:02.098799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.098768    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f288de4-2jbbf"]
Nov 02 02:20:02.617904 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.617868    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"]
Nov 02 02:20:02.618403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.618388    4912 cpu_manager.go:414] "containerMap: removing container" podUID="319b3f4a-962f-4eae-b2fa-3facce48ec17" containerName="revoke-keys"
Nov 02 02:20:02.618455 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.618405    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319b3f4a-962f-4eae-b2fa-3facce48ec17" containerName="revoke-keys"
Nov 02 02:20:02.618529 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.618519    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="319b3f4a-962f-4eae-b2fa-3facce48ec17" containerName="revoke-keys"
Nov 02 02:20:02.621832 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.621814    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.648106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.648039    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xbnn\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.648259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.648179    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsz2\" (UniqueName: \"kubernetes.io/projected/6877d44a-d9d4-4abd-ad99-754ea7ffe480-kube-api-access-vwsz2\") pod \"maas-api-revoke-sub-2e1f5b5f-9xbnn\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.648303 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.648258    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xbnn\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.676675 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.676646    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:20:02.726246 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.726212    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:20:02.729084 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.729046    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"]
Nov 02 02:20:02.749728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.749703    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xbnn\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.749842 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.749763    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsz2\" (UniqueName: \"kubernetes.io/projected/6877d44a-d9d4-4abd-ad99-754ea7ffe480-kube-api-access-vwsz2\") pod \"maas-api-revoke-sub-2e1f5b5f-9xbnn\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.749842 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.749831    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xbnn\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.750436 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.750399    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xbnn\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.752399 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.752377    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xbnn\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.810795 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.810766    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsz2\" (UniqueName: \"kubernetes.io/projected/6877d44a-d9d4-4abd-ad99-754ea7ffe480-kube-api-access-vwsz2\") pod \"maas-api-revoke-sub-2e1f5b5f-9xbnn\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:02.934605 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:02.934517    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:03.225093 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:03.224986    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4daede-23e0-4199-834d-1a0ecf9cb72c" path="/var/lib/kubelet/pods/5f4daede-23e0-4199-834d-1a0ecf9cb72c/volumes"
Nov 02 02:20:03.310251 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:03.310224    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"]
Nov 02 02:20:03.311893 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:20:03.311868    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6877d44a_d9d4_4abd_ad99_754ea7ffe480.slice/crio-d5155e908d910b6e3a6fb1f712179e9553933b717ee841e34ff430f8891075bb WatchSource:0}: Error finding container d5155e908d910b6e3a6fb1f712179e9553933b717ee841e34ff430f8891075bb: Status 404 returned error can't find the container with id d5155e908d910b6e3a6fb1f712179e9553933b717ee841e34ff430f8891075bb
Nov 02 02:20:03.719348 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:03.719307    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn" event={"ID":"6877d44a-d9d4-4abd-ad99-754ea7ffe480","Type":"ContainerStarted","Data":"10180aca3230887c34db14e142707bf6dbd4ba4349614adb0bbfd4fc4856ae09"}
Nov 02 02:20:03.719559 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:03.719374    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn" event={"ID":"6877d44a-d9d4-4abd-ad99-754ea7ffe480","Type":"ContainerStarted","Data":"d5155e908d910b6e3a6fb1f712179e9553933b717ee841e34ff430f8891075bb"}
Nov 02 02:20:03.834949 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:03.834848    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn" podStartSLOduration=1.834834157 podStartE2EDuration="1.834834157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:20:03.833569304 +0000 UTC m=+2896.739034661" watchObservedRunningTime="2026-11-02 02:20:03.834834157 +0000 UTC m=+2896.740299560"
Nov 02 02:20:04.726777 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:04.726745    4912 generic.go:356] "Generic (PLEG): container finished" podID="6877d44a-d9d4-4abd-ad99-754ea7ffe480" containerID="10180aca3230887c34db14e142707bf6dbd4ba4349614adb0bbfd4fc4856ae09" exitCode=0
Nov 02 02:20:04.727251 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:04.726835    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn" event={"ID":"6877d44a-d9d4-4abd-ad99-754ea7ffe480","Type":"ContainerDied","Data":"10180aca3230887c34db14e142707bf6dbd4ba4349614adb0bbfd4fc4856ae09"}
Nov 02 02:20:05.875375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:05.875348    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:05.983125 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:05.983080    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6877d44a-d9d4-4abd-ad99-754ea7ffe480-kube-api-access-vwsz2\" (UniqueName: \"kubernetes.io/projected/6877d44a-d9d4-4abd-ad99-754ea7ffe480-kube-api-access-vwsz2\") pod \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") "
Nov 02 02:20:05.983354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:05.983261    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-openshift-ca\") pod \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") "
Nov 02 02:20:05.983354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:05.983283    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-xks-ca\") pod \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\" (UID: \"6877d44a-d9d4-4abd-ad99-754ea7ffe480\") "
Nov 02 02:20:05.983647 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:05.983624    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-openshift-ca" pod "6877d44a-d9d4-4abd-ad99-754ea7ffe480" (UID: "6877d44a-d9d4-4abd-ad99-754ea7ffe480"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:20:05.985355 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:05.985335    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6877d44a-d9d4-4abd-ad99-754ea7ffe480-kube-api-access-vwsz2" pod "6877d44a-d9d4-4abd-ad99-754ea7ffe480" (UID: "6877d44a-d9d4-4abd-ad99-754ea7ffe480"). InnerVolumeSpecName "kube-api-access-vwsz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:20:05.985527 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:05.985507    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-xks-ca" pod "6877d44a-d9d4-4abd-ad99-754ea7ffe480" (UID: "6877d44a-d9d4-4abd-ad99-754ea7ffe480"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:20:06.084952 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:06.084917    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:06.084952 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:06.084947    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6877d44a-d9d4-4abd-ad99-754ea7ffe480-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:06.084952 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:06.084957    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwsz2\" (UniqueName: \"kubernetes.io/projected/6877d44a-d9d4-4abd-ad99-754ea7ffe480-kube-api-access-vwsz2\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:06.739947 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:06.739909    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"
Nov 02 02:20:06.740155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:06.739940    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn" event={"ID":"6877d44a-d9d4-4abd-ad99-754ea7ffe480","Type":"ContainerDied","Data":"d5155e908d910b6e3a6fb1f712179e9553933b717ee841e34ff430f8891075bb"}
Nov 02 02:20:06.740155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:06.740017    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5155e908d910b6e3a6fb1f712179e9553933b717ee841e34ff430f8891075bb"
Nov 02 02:20:08.635415 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:08.635371    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:20:08.817628 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:08.817591    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:20:12.137654 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:20:12.137615    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:20:15.114787 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.114752    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"]
Nov 02 02:20:15.115378 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.115350    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6877d44a-d9d4-4abd-ad99-754ea7ffe480" containerName="revoke-keys"
Nov 02 02:20:15.115378 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.115380    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6877d44a-d9d4-4abd-ad99-754ea7ffe480" containerName="revoke-keys"
Nov 02 02:20:15.115485 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.115462    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6877d44a-d9d4-4abd-ad99-754ea7ffe480" containerName="revoke-keys"
Nov 02 02:20:15.118680 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.118660    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.171852 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.171801    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:20:15.221182 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.221152    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:20:15.225914 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.225888    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"]
Nov 02 02:20:15.281846 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.281805    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmjw\" (UniqueName: \"kubernetes.io/projected/045aa839-475e-4fd8-935a-e7dd48c0f8d8-kube-api-access-4tmjw\") pod \"maas-api-revoke-sub-ec843018-clndb\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.282086 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.281957    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-clndb\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.282197 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.282124    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-clndb\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.382807 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.382722    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmjw\" (UniqueName: \"kubernetes.io/projected/045aa839-475e-4fd8-935a-e7dd48c0f8d8-kube-api-access-4tmjw\") pod \"maas-api-revoke-sub-ec843018-clndb\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.382807 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.382787    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-clndb\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.383008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.382809    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-clndb\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.383536 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.383500    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-clndb\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.385375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.385352    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-clndb\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.443627 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.443594    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmjw\" (UniqueName: \"kubernetes.io/projected/045aa839-475e-4fd8-935a-e7dd48c0f8d8-kube-api-access-4tmjw\") pod \"maas-api-revoke-sub-ec843018-clndb\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:15.731882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:15.731806    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:16.105757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:16.105730    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"]
Nov 02 02:20:16.107387 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:20:16.107358    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045aa839_475e_4fd8_935a_e7dd48c0f8d8.slice/crio-dc7417e98514f5d99df40066543403ec0c6066ec9cacfb1ce4dd0f44b6096974 WatchSource:0}: Error finding container dc7417e98514f5d99df40066543403ec0c6066ec9cacfb1ce4dd0f44b6096974: Status 404 returned error can't find the container with id dc7417e98514f5d99df40066543403ec0c6066ec9cacfb1ce4dd0f44b6096974
Nov 02 02:20:16.806553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:16.806510    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb" event={"ID":"045aa839-475e-4fd8-935a-e7dd48c0f8d8","Type":"ContainerStarted","Data":"d3d60ae9d084e1610207e98c80ac33acd79758faf63fc474d8841190f636e23d"}
Nov 02 02:20:16.806967 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:16.806609    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb" event={"ID":"045aa839-475e-4fd8-935a-e7dd48c0f8d8","Type":"ContainerStarted","Data":"dc7417e98514f5d99df40066543403ec0c6066ec9cacfb1ce4dd0f44b6096974"}
Nov 02 02:20:16.922008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:16.921958    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb" podStartSLOduration=1.921945144 podStartE2EDuration="1.921945144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:20:16.920732939 +0000 UTC m=+2909.826198297" watchObservedRunningTime="2026-11-02 02:20:16.921945144 +0000 UTC m=+2909.827410500"
Nov 02 02:20:17.813536 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:17.813503    4912 generic.go:356] "Generic (PLEG): container finished" podID="045aa839-475e-4fd8-935a-e7dd48c0f8d8" containerID="d3d60ae9d084e1610207e98c80ac33acd79758faf63fc474d8841190f636e23d" exitCode=0
Nov 02 02:20:17.814026 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:17.813598    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb" event={"ID":"045aa839-475e-4fd8-935a-e7dd48c0f8d8","Type":"ContainerDied","Data":"d3d60ae9d084e1610207e98c80ac33acd79758faf63fc474d8841190f636e23d"}
Nov 02 02:20:18.963331 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:18.963306    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:19.120735 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.120708    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-xks-ca\") pod \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") "
Nov 02 02:20:19.120904 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.120785    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-openshift-ca\") pod \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") "
Nov 02 02:20:19.120971 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.120953    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045aa839-475e-4fd8-935a-e7dd48c0f8d8-kube-api-access-4tmjw\" (UniqueName: \"kubernetes.io/projected/045aa839-475e-4fd8-935a-e7dd48c0f8d8-kube-api-access-4tmjw\") pod \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\" (UID: \"045aa839-475e-4fd8-935a-e7dd48c0f8d8\") "
Nov 02 02:20:19.121144 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.121118    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-openshift-ca" pod "045aa839-475e-4fd8-935a-e7dd48c0f8d8" (UID: "045aa839-475e-4fd8-935a-e7dd48c0f8d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:20:19.121321 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.121303    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:19.123138 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.123111    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-xks-ca" pod "045aa839-475e-4fd8-935a-e7dd48c0f8d8" (UID: "045aa839-475e-4fd8-935a-e7dd48c0f8d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:20:19.123243 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.123163    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045aa839-475e-4fd8-935a-e7dd48c0f8d8-kube-api-access-4tmjw" pod "045aa839-475e-4fd8-935a-e7dd48c0f8d8" (UID: "045aa839-475e-4fd8-935a-e7dd48c0f8d8"). InnerVolumeSpecName "kube-api-access-4tmjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:20:19.222703 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.222667    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmjw\" (UniqueName: \"kubernetes.io/projected/045aa839-475e-4fd8-935a-e7dd48c0f8d8-kube-api-access-4tmjw\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:19.222703 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.222701    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045aa839-475e-4fd8-935a-e7dd48c0f8d8-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:19.827255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.827229    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"
Nov 02 02:20:19.827255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.827252    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb" event={"ID":"045aa839-475e-4fd8-935a-e7dd48c0f8d8","Type":"ContainerDied","Data":"dc7417e98514f5d99df40066543403ec0c6066ec9cacfb1ce4dd0f44b6096974"}
Nov 02 02:20:19.827482 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:19.827304    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc7417e98514f5d99df40066543403ec0c6066ec9cacfb1ce4dd0f44b6096974"
Nov 02 02:20:37.952354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:37.952258    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:20:38.134701 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:38.134655    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:20:42.140948 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:20:42.140909    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:20:48.324387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:48.324352    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:20:50.666498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.666466    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"]
Nov 02 02:20:50.667001 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.666986    4912 cpu_manager.go:414] "containerMap: removing container" podUID="045aa839-475e-4fd8-935a-e7dd48c0f8d8" containerName="revoke-keys"
Nov 02 02:20:50.667117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.667003    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045aa839-475e-4fd8-935a-e7dd48c0f8d8" containerName="revoke-keys"
Nov 02 02:20:50.667117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.667093    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="045aa839-475e-4fd8-935a-e7dd48c0f8d8" containerName="revoke-keys"
Nov 02 02:20:50.670341 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.670327    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.724167 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.724141    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:20:50.773821 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.773787    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:20:50.776764 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.776742    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"]
Nov 02 02:20:50.793372 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.793340    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b55kj\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.793515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.793478    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfd9q\" (UniqueName: \"kubernetes.io/projected/b7d6f675-a15c-49ab-a9a2-deb923b8e974-kube-api-access-jfd9q\") pod \"maas-api-revoke-sub-046e1343-b55kj\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.793587 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.793570    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b55kj\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.894824 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.894788    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfd9q\" (UniqueName: \"kubernetes.io/projected/b7d6f675-a15c-49ab-a9a2-deb923b8e974-kube-api-access-jfd9q\") pod \"maas-api-revoke-sub-046e1343-b55kj\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.895002 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.894840    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b55kj\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.895002 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.894877    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b55kj\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.895634 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.895615    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b55kj\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.897508 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.897483    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b55kj\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.955882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.955807    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfd9q\" (UniqueName: \"kubernetes.io/projected/b7d6f675-a15c-49ab-a9a2-deb923b8e974-kube-api-access-jfd9q\") pod \"maas-api-revoke-sub-046e1343-b55kj\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:50.986586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:50.986555    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:51.361902 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:51.361872    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"]
Nov 02 02:20:51.363168 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:20:51.363129    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d6f675_a15c_49ab_a9a2_deb923b8e974.slice/crio-3874642b9b7f1894be9b013ee26cd0fcaa5053a53a1425aef20a6c1ab33796d3 WatchSource:0}: Error finding container 3874642b9b7f1894be9b013ee26cd0fcaa5053a53a1425aef20a6c1ab33796d3: Status 404 returned error can't find the container with id 3874642b9b7f1894be9b013ee26cd0fcaa5053a53a1425aef20a6c1ab33796d3
Nov 02 02:20:52.046088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:52.045974    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj" event={"ID":"b7d6f675-a15c-49ab-a9a2-deb923b8e974","Type":"ContainerStarted","Data":"a0f7ff7d113ee9cc898a3e141dc27b2476c12b67dd2373e575b52483cb512d6d"}
Nov 02 02:20:52.046088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:52.046039    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj" event={"ID":"b7d6f675-a15c-49ab-a9a2-deb923b8e974","Type":"ContainerStarted","Data":"3874642b9b7f1894be9b013ee26cd0fcaa5053a53a1425aef20a6c1ab33796d3"}
Nov 02 02:20:52.162049 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:52.161996    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj" podStartSLOduration=2.161980587 podStartE2EDuration="2.161980587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:20:52.160476802 +0000 UTC m=+2945.065942160" watchObservedRunningTime="2026-11-02 02:20:52.161980587 +0000 UTC m=+2945.067445943"
Nov 02 02:20:53.054337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:53.054302    4912 generic.go:356] "Generic (PLEG): container finished" podID="b7d6f675-a15c-49ab-a9a2-deb923b8e974" containerID="a0f7ff7d113ee9cc898a3e141dc27b2476c12b67dd2373e575b52483cb512d6d" exitCode=0
Nov 02 02:20:53.054816 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:53.054351    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj" event={"ID":"b7d6f675-a15c-49ab-a9a2-deb923b8e974","Type":"ContainerDied","Data":"a0f7ff7d113ee9cc898a3e141dc27b2476c12b67dd2373e575b52483cb512d6d"}
Nov 02 02:20:54.202124 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.202100    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:54.331296 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.331207    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-xks-ca\") pod \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") "
Nov 02 02:20:54.331486 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.331325    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-openshift-ca\") pod \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") "
Nov 02 02:20:54.331486 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.331407    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d6f675-a15c-49ab-a9a2-deb923b8e974-kube-api-access-jfd9q\" (UniqueName: \"kubernetes.io/projected/b7d6f675-a15c-49ab-a9a2-deb923b8e974-kube-api-access-jfd9q\") pod \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\" (UID: \"b7d6f675-a15c-49ab-a9a2-deb923b8e974\") "
Nov 02 02:20:54.331765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.331737    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-openshift-ca" pod "b7d6f675-a15c-49ab-a9a2-deb923b8e974" (UID: "b7d6f675-a15c-49ab-a9a2-deb923b8e974"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:20:54.333633 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.333604    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-xks-ca" pod "b7d6f675-a15c-49ab-a9a2-deb923b8e974" (UID: "b7d6f675-a15c-49ab-a9a2-deb923b8e974"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:20:54.333752 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.333707    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d6f675-a15c-49ab-a9a2-deb923b8e974-kube-api-access-jfd9q" pod "b7d6f675-a15c-49ab-a9a2-deb923b8e974" (UID: "b7d6f675-a15c-49ab-a9a2-deb923b8e974"). InnerVolumeSpecName "kube-api-access-jfd9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:20:54.432645 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.432613    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:54.432645 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.432640    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfd9q\" (UniqueName: \"kubernetes.io/projected/b7d6f675-a15c-49ab-a9a2-deb923b8e974-kube-api-access-jfd9q\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:54.432645 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.432650    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6f675-a15c-49ab-a9a2-deb923b8e974-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:20:54.781603 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:54.781566    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:20:55.068342 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:55.068258    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"
Nov 02 02:20:55.068342 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:55.068285    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj" event={"ID":"b7d6f675-a15c-49ab-a9a2-deb923b8e974","Type":"ContainerDied","Data":"3874642b9b7f1894be9b013ee26cd0fcaa5053a53a1425aef20a6c1ab33796d3"}
Nov 02 02:20:55.068342 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:20:55.068328    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3874642b9b7f1894be9b013ee26cd0fcaa5053a53a1425aef20a6c1ab33796d3"
Nov 02 02:21:01.415179 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.415133    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"]
Nov 02 02:21:01.415728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.415701    4912 cpu_manager.go:414] "containerMap: removing container" podUID="b7d6f675-a15c-49ab-a9a2-deb923b8e974" containerName="revoke-keys"
Nov 02 02:21:01.415728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.415717    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d6f675-a15c-49ab-a9a2-deb923b8e974" containerName="revoke-keys"
Nov 02 02:21:01.415848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.415783    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d6f675-a15c-49ab-a9a2-deb923b8e974" containerName="revoke-keys"
Nov 02 02:21:01.418940 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.418917    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.477541 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.477501    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"]
Nov 02 02:21:01.504292 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.504259    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vn9h2\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.504515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.504449    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdt9\" (UniqueName: \"kubernetes.io/projected/fcbeda3e-c4bf-420d-973a-229374d6658b-kube-api-access-vgdt9\") pod \"maas-api-revoke-sub-d3fba056-vn9h2\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.504583 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.504524    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vn9h2\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.521922 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.521891    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:21:01.522090 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.521963    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:21:01.605597 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.605562    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdt9\" (UniqueName: \"kubernetes.io/projected/fcbeda3e-c4bf-420d-973a-229374d6658b-kube-api-access-vgdt9\") pod \"maas-api-revoke-sub-d3fba056-vn9h2\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.605823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.605636    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vn9h2\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.605823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.605697    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vn9h2\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.606322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.606296    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vn9h2\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.608491 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.608465    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vn9h2\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.666010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.665935    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdt9\" (UniqueName: \"kubernetes.io/projected/fcbeda3e-c4bf-420d-973a-229374d6658b-kube-api-access-vgdt9\") pod \"maas-api-revoke-sub-d3fba056-vn9h2\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:01.732119 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:01.732080    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:02.111800 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:02.111770    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"]
Nov 02 02:21:02.111946 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:21:02.111880    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbeda3e_c4bf_420d_973a_229374d6658b.slice/crio-17629c1568270c46f15ab29f31f84497b89a7fc0c2078834b70096986337d4df WatchSource:0}: Error finding container 17629c1568270c46f15ab29f31f84497b89a7fc0c2078834b70096986337d4df: Status 404 returned error can't find the container with id 17629c1568270c46f15ab29f31f84497b89a7fc0c2078834b70096986337d4df
Nov 02 02:21:03.124733 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:03.124700    4912 generic.go:356] "Generic (PLEG): container finished" podID="fcbeda3e-c4bf-420d-973a-229374d6658b" containerID="a3738dfe481e3482fd5400b2fe7328993f11b1033c340bf95ba1b3756192c67b" exitCode=0
Nov 02 02:21:03.125197 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:03.124795    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2" event={"ID":"fcbeda3e-c4bf-420d-973a-229374d6658b","Type":"ContainerDied","Data":"a3738dfe481e3482fd5400b2fe7328993f11b1033c340bf95ba1b3756192c67b"}
Nov 02 02:21:03.125197 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:03.124862    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2" event={"ID":"fcbeda3e-c4bf-420d-973a-229374d6658b","Type":"ContainerStarted","Data":"17629c1568270c46f15ab29f31f84497b89a7fc0c2078834b70096986337d4df"}
Nov 02 02:21:04.336909 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.336878    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:04.434453 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.434343    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbeda3e-c4bf-420d-973a-229374d6658b-kube-api-access-vgdt9\" (UniqueName: \"kubernetes.io/projected/fcbeda3e-c4bf-420d-973a-229374d6658b-kube-api-access-vgdt9\") pod \"fcbeda3e-c4bf-420d-973a-229374d6658b\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") "
Nov 02 02:21:04.434636 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.434538    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-xks-ca\") pod \"fcbeda3e-c4bf-420d-973a-229374d6658b\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") "
Nov 02 02:21:04.434636 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.434616    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-openshift-ca\") pod \"fcbeda3e-c4bf-420d-973a-229374d6658b\" (UID: \"fcbeda3e-c4bf-420d-973a-229374d6658b\") "
Nov 02 02:21:04.435282 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.435250    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-openshift-ca" pod "fcbeda3e-c4bf-420d-973a-229374d6658b" (UID: "fcbeda3e-c4bf-420d-973a-229374d6658b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:21:04.438225 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.438198    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-xks-ca" pod "fcbeda3e-c4bf-420d-973a-229374d6658b" (UID: "fcbeda3e-c4bf-420d-973a-229374d6658b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:21:04.438347 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.438234    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbeda3e-c4bf-420d-973a-229374d6658b-kube-api-access-vgdt9" pod "fcbeda3e-c4bf-420d-973a-229374d6658b" (UID: "fcbeda3e-c4bf-420d-973a-229374d6658b"). InnerVolumeSpecName "kube-api-access-vgdt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:21:04.536014 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.535976    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:04.536014 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.536015    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbeda3e-c4bf-420d-973a-229374d6658b-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:04.536325 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:04.536031    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgdt9\" (UniqueName: \"kubernetes.io/projected/fcbeda3e-c4bf-420d-973a-229374d6658b-kube-api-access-vgdt9\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:05.143237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:05.143204    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"
Nov 02 02:21:05.143237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:05.143227    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2" event={"ID":"fcbeda3e-c4bf-420d-973a-229374d6658b","Type":"ContainerDied","Data":"17629c1568270c46f15ab29f31f84497b89a7fc0c2078834b70096986337d4df"}
Nov 02 02:21:05.143491 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:05.143273    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17629c1568270c46f15ab29f31f84497b89a7fc0c2078834b70096986337d4df"
Nov 02 02:21:08.485029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:08.484993    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:21:08.669484 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:08.669441    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:21:12.139540 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:21:12.139499    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:21:39.312841 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.312802    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"]
Nov 02 02:21:39.313412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.313292    4912 cpu_manager.go:414] "containerMap: removing container" podUID="fcbeda3e-c4bf-420d-973a-229374d6658b" containerName="revoke-keys"
Nov 02 02:21:39.313412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.313304    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbeda3e-c4bf-420d-973a-229374d6658b" containerName="revoke-keys"
Nov 02 02:21:39.313412 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.313370    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcbeda3e-c4bf-420d-973a-229374d6658b" containerName="revoke-keys"
Nov 02 02:21:39.316787 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.316767    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.369247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.369213    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:21:39.405146 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.405113    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kg4d\" (UniqueName: \"kubernetes.io/projected/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-kube-api-access-2kg4d\") pod \"maas-api-revoke-sub-ae594979-78n7t\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.405316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.405186    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-78n7t\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.405365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.405345    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-78n7t\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.418773 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.418743    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:21:39.421882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.421861    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"]
Nov 02 02:21:39.506554 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.506525    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-78n7t\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.506711 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.506575    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kg4d\" (UniqueName: \"kubernetes.io/projected/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-kube-api-access-2kg4d\") pod \"maas-api-revoke-sub-ae594979-78n7t\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.506711 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.506633    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-78n7t\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.507331 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.507312    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-78n7t\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.509042 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.509019    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-78n7t\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.567420 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.567350    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kg4d\" (UniqueName: \"kubernetes.io/projected/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-kube-api-access-2kg4d\") pod \"maas-api-revoke-sub-ae594979-78n7t\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:39.630115 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:39.630082    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:40.008954 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:40.008922    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"]
Nov 02 02:21:40.009258 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:21:40.009236    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cc28ce_af4c_4c6f_9cf5_3ad94cb3d368.slice/crio-c52fc1bd832106d711ed726f28d4b1c098f5bdb80fefbfa498d50256d7d54bd5 WatchSource:0}: Error finding container c52fc1bd832106d711ed726f28d4b1c098f5bdb80fefbfa498d50256d7d54bd5: Status 404 returned error can't find the container with id c52fc1bd832106d711ed726f28d4b1c098f5bdb80fefbfa498d50256d7d54bd5
Nov 02 02:21:40.386300 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:40.386257    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t" event={"ID":"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368","Type":"ContainerStarted","Data":"a0dbc2369eefcda644d2efa1b6bca40aa1880ed7d4e51ec63bf9c67bc0af99d8"}
Nov 02 02:21:40.386714 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:40.386317    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t" event={"ID":"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368","Type":"ContainerStarted","Data":"c52fc1bd832106d711ed726f28d4b1c098f5bdb80fefbfa498d50256d7d54bd5"}
Nov 02 02:21:40.501939 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:40.501883    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t" podStartSLOduration=1.501867533 podStartE2EDuration="1.501867533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:21:40.501223399 +0000 UTC m=+2993.406688775" watchObservedRunningTime="2026-11-02 02:21:40.501867533 +0000 UTC m=+2993.407332889"
Nov 02 02:21:41.393781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:41.393750    4912 generic.go:356] "Generic (PLEG): container finished" podID="62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368" containerID="a0dbc2369eefcda644d2efa1b6bca40aa1880ed7d4e51ec63bf9c67bc0af99d8" exitCode=0
Nov 02 02:21:41.394232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:41.393831    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t" event={"ID":"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368","Type":"ContainerDied","Data":"a0dbc2369eefcda644d2efa1b6bca40aa1880ed7d4e51ec63bf9c67bc0af99d8"}
Nov 02 02:21:42.137718 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:21:42.137681    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:21:42.541069 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.541027    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:42.634722 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.634684    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-openshift-ca\") pod \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") "
Nov 02 02:21:42.634929 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.634743    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-xks-ca\") pod \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") "
Nov 02 02:21:42.634929 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.634770    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-kube-api-access-2kg4d\" (UniqueName: \"kubernetes.io/projected/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-kube-api-access-2kg4d\") pod \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\" (UID: \"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368\") "
Nov 02 02:21:42.635101 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.635040    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-openshift-ca" pod "62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368" (UID: "62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:21:42.637007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.636983    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-xks-ca" pod "62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368" (UID: "62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:21:42.637163 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.637052    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-kube-api-access-2kg4d" pod "62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368" (UID: "62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368"). InnerVolumeSpecName "kube-api-access-2kg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:21:42.736330 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.736244    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:42.736330 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.736278    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:42.736330 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:42.736288    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kg4d\" (UniqueName: \"kubernetes.io/projected/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368-kube-api-access-2kg4d\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:21:43.414842 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:43.414795    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t" event={"ID":"62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368","Type":"ContainerDied","Data":"c52fc1bd832106d711ed726f28d4b1c098f5bdb80fefbfa498d50256d7d54bd5"}
Nov 02 02:21:43.414842 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:43.414835    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"
Nov 02 02:21:43.415084 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:43.414889    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c52fc1bd832106d711ed726f28d4b1c098f5bdb80fefbfa498d50256d7d54bd5"
Nov 02 02:21:47.874249 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:47.874224    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:21:47.880982 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:47.880961    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:21:48.021550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:48.021515    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:21:48.029727 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:48.029703    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:21:49.623122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:49.623087    4912 scope.go:122] "RemoveContainer" containerID="e7e9d63b502c7b619c20b2b390817343dbb1d12cff66c779e8a65f7a2b892991"
Nov 02 02:21:55.159232 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:21:55.159196    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:22:10.514755 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.514668    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"]
Nov 02 02:22:10.515291 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.515153    4912 cpu_manager.go:414] "containerMap: removing container" podUID="62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368" containerName="revoke-keys"
Nov 02 02:22:10.515291 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.515165    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368" containerName="revoke-keys"
Nov 02 02:22:10.515291 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.515259    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368" containerName="revoke-keys"
Nov 02 02:22:10.519935 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.519918    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.578198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.578168    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"]
Nov 02 02:22:10.623774 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.623737    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:22:10.623774 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.623778    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:22:10.648440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.648412    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lp7xl\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.648609 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.648500    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lp7xl\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.648609 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.648544    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt2tp\" (UniqueName: \"kubernetes.io/projected/e16d22bc-876f-48b8-8fc3-5d4d133f354d-kube-api-access-qt2tp\") pod \"maas-api-revoke-sub-a4aef7d7-lp7xl\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.749403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.749367    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lp7xl\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.749613 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.749433    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lp7xl\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.749613 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.749470    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2tp\" (UniqueName: \"kubernetes.io/projected/e16d22bc-876f-48b8-8fc3-5d4d133f354d-kube-api-access-qt2tp\") pod \"maas-api-revoke-sub-a4aef7d7-lp7xl\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.750044 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.750020    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lp7xl\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.751951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.751930    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lp7xl\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.810213 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.810137    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2tp\" (UniqueName: \"kubernetes.io/projected/e16d22bc-876f-48b8-8fc3-5d4d133f354d-kube-api-access-qt2tp\") pod \"maas-api-revoke-sub-a4aef7d7-lp7xl\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:10.833667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:10.833639    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:11.208558 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:11.208528    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"]
Nov 02 02:22:11.209647 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:22:11.209622    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16d22bc_876f_48b8_8fc3_5d4d133f354d.slice/crio-f145a92f0c2d8a9e9d05e1ae23452c304b93c017c5851b82e274138e51abb6d5 WatchSource:0}: Error finding container f145a92f0c2d8a9e9d05e1ae23452c304b93c017c5851b82e274138e51abb6d5: Status 404 returned error can't find the container with id f145a92f0c2d8a9e9d05e1ae23452c304b93c017c5851b82e274138e51abb6d5
Nov 02 02:22:11.609092 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:11.609029    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl" event={"ID":"e16d22bc-876f-48b8-8fc3-5d4d133f354d","Type":"ContainerStarted","Data":"cd3cb112e79523f3d5d93a9f90b31385b3b55861a9840ac2e47b0cc5f75c52d4"}
Nov 02 02:22:11.609501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:11.609116    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl" event={"ID":"e16d22bc-876f-48b8-8fc3-5d4d133f354d","Type":"ContainerStarted","Data":"f145a92f0c2d8a9e9d05e1ae23452c304b93c017c5851b82e274138e51abb6d5"}
Nov 02 02:22:11.724435 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:11.724334    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl" podStartSLOduration=1.724320306 podStartE2EDuration="1.724320306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:22:11.723793022 +0000 UTC m=+3024.629258401" watchObservedRunningTime="2026-11-02 02:22:11.724320306 +0000 UTC m=+3024.629785663"
Nov 02 02:22:12.137529 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:22:12.137481    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:22:12.616926 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:12.616888    4912 generic.go:356] "Generic (PLEG): container finished" podID="e16d22bc-876f-48b8-8fc3-5d4d133f354d" containerID="cd3cb112e79523f3d5d93a9f90b31385b3b55861a9840ac2e47b0cc5f75c52d4" exitCode=0
Nov 02 02:22:12.617392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:12.616975    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl" event={"ID":"e16d22bc-876f-48b8-8fc3-5d4d133f354d","Type":"ContainerDied","Data":"cd3cb112e79523f3d5d93a9f90b31385b3b55861a9840ac2e47b0cc5f75c52d4"}
Nov 02 02:22:13.769991 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.769965    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:13.881806 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.881714    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-xks-ca\") pod \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") "
Nov 02 02:22:13.881806 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.881792    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16d22bc-876f-48b8-8fc3-5d4d133f354d-kube-api-access-qt2tp\" (UniqueName: \"kubernetes.io/projected/e16d22bc-876f-48b8-8fc3-5d4d133f354d-kube-api-access-qt2tp\") pod \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") "
Nov 02 02:22:13.882110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.881878    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-openshift-ca\") pod \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\" (UID: \"e16d22bc-876f-48b8-8fc3-5d4d133f354d\") "
Nov 02 02:22:13.882272 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.882248    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-openshift-ca" pod "e16d22bc-876f-48b8-8fc3-5d4d133f354d" (UID: "e16d22bc-876f-48b8-8fc3-5d4d133f354d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:22:13.884248 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.884220    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-xks-ca" pod "e16d22bc-876f-48b8-8fc3-5d4d133f354d" (UID: "e16d22bc-876f-48b8-8fc3-5d4d133f354d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:22:13.884752 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.884352    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16d22bc-876f-48b8-8fc3-5d4d133f354d-kube-api-access-qt2tp" pod "e16d22bc-876f-48b8-8fc3-5d4d133f354d" (UID: "e16d22bc-876f-48b8-8fc3-5d4d133f354d"). InnerVolumeSpecName "kube-api-access-qt2tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:22:13.982804 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.982766    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:22:13.982804 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.982798    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt2tp\" (UniqueName: \"kubernetes.io/projected/e16d22bc-876f-48b8-8fc3-5d4d133f354d-kube-api-access-qt2tp\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:22:13.982804 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:13.982810    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e16d22bc-876f-48b8-8fc3-5d4d133f354d-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:22:14.630928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:14.630882    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl" event={"ID":"e16d22bc-876f-48b8-8fc3-5d4d133f354d","Type":"ContainerDied","Data":"f145a92f0c2d8a9e9d05e1ae23452c304b93c017c5851b82e274138e51abb6d5"}
Nov 02 02:22:14.630928 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:14.630925    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"
Nov 02 02:22:14.631238 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:14.630949    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f145a92f0c2d8a9e9d05e1ae23452c304b93c017c5851b82e274138e51abb6d5"
Nov 02 02:22:31.054970 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:31.054930    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"]
Nov 02 02:22:31.104618 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:31.104581    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72aa13a4-2f5l6"]
Nov 02 02:22:31.225888 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:31.225846    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1" path="/var/lib/kubelet/pods/6b847a4d-9f5a-4e28-9972-d2bbc5dc01b1/volumes"
Nov 02 02:22:41.202253 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.202208    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"]
Nov 02 02:22:41.202684 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.202670    4912 cpu_manager.go:414] "containerMap: removing container" podUID="e16d22bc-876f-48b8-8fc3-5d4d133f354d" containerName="revoke-keys"
Nov 02 02:22:41.202684 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.202682    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16d22bc-876f-48b8-8fc3-5d4d133f354d" containerName="revoke-keys"
Nov 02 02:22:41.202793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.202782    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16d22bc-876f-48b8-8fc3-5d4d133f354d" containerName="revoke-keys"
Nov 02 02:22:41.205769 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.205747    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.239472 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.239438    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtt78\" (UniqueName: \"kubernetes.io/projected/03922de4-1b33-4bd5-9a52-619a88b269b9-kube-api-access-vtt78\") pod \"maas-api-revoke-sub-79805ca9-86h57\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.239667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.239573    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-86h57\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.239667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.239596    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-86h57\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.260440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.260409    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:22:41.309730 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.309698    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:22:41.312911 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.312887    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"]
Nov 02 02:22:41.340255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.340222    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-86h57\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.340255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.340256    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-86h57\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.340607 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.340356    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtt78\" (UniqueName: \"kubernetes.io/projected/03922de4-1b33-4bd5-9a52-619a88b269b9-kube-api-access-vtt78\") pod \"maas-api-revoke-sub-79805ca9-86h57\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.340888 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.340864    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-86h57\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.342820 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.342795    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-86h57\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.401247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.401217    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtt78\" (UniqueName: \"kubernetes.io/projected/03922de4-1b33-4bd5-9a52-619a88b269b9-kube-api-access-vtt78\") pod \"maas-api-revoke-sub-79805ca9-86h57\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.519034 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.518955    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:41.894156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:41.894131    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"]
Nov 02 02:22:41.895421 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:22:41.895388    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03922de4_1b33_4bd5_9a52_619a88b269b9.slice/crio-fdad843ea5070cd1fbaee31f9a424603a593ab49fb3b632f9a4cfa452f90d4f8 WatchSource:0}: Error finding container fdad843ea5070cd1fbaee31f9a424603a593ab49fb3b632f9a4cfa452f90d4f8: Status 404 returned error can't find the container with id fdad843ea5070cd1fbaee31f9a424603a593ab49fb3b632f9a4cfa452f90d4f8
Nov 02 02:22:42.137568 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:22:42.137531    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:22:42.806893 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:42.806854    4912 generic.go:356] "Generic (PLEG): container finished" podID="03922de4-1b33-4bd5-9a52-619a88b269b9" containerID="001cf04aed7dc66b617415420b2cb81b32a14d1ae144abbe0bd7f5087cbcb873" exitCode=0
Nov 02 02:22:42.807312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:42.806899    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57" event={"ID":"03922de4-1b33-4bd5-9a52-619a88b269b9","Type":"ContainerDied","Data":"001cf04aed7dc66b617415420b2cb81b32a14d1ae144abbe0bd7f5087cbcb873"}
Nov 02 02:22:42.807312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:42.806980    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57" event={"ID":"03922de4-1b33-4bd5-9a52-619a88b269b9","Type":"ContainerStarted","Data":"fdad843ea5070cd1fbaee31f9a424603a593ab49fb3b632f9a4cfa452f90d4f8"}
Nov 02 02:22:43.957722 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:43.957691    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:44.068231 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.068137    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-xks-ca\") pod \"03922de4-1b33-4bd5-9a52-619a88b269b9\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") "
Nov 02 02:22:44.068231 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.068197    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-openshift-ca\") pod \"03922de4-1b33-4bd5-9a52-619a88b269b9\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") "
Nov 02 02:22:44.068487 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.068272    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03922de4-1b33-4bd5-9a52-619a88b269b9-kube-api-access-vtt78\" (UniqueName: \"kubernetes.io/projected/03922de4-1b33-4bd5-9a52-619a88b269b9-kube-api-access-vtt78\") pod \"03922de4-1b33-4bd5-9a52-619a88b269b9\" (UID: \"03922de4-1b33-4bd5-9a52-619a88b269b9\") "
Nov 02 02:22:44.068609 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.068569    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-openshift-ca" pod "03922de4-1b33-4bd5-9a52-619a88b269b9" (UID: "03922de4-1b33-4bd5-9a52-619a88b269b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:22:44.070534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.070505    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-xks-ca" pod "03922de4-1b33-4bd5-9a52-619a88b269b9" (UID: "03922de4-1b33-4bd5-9a52-619a88b269b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:22:44.070534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.070516    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03922de4-1b33-4bd5-9a52-619a88b269b9-kube-api-access-vtt78" pod "03922de4-1b33-4bd5-9a52-619a88b269b9" (UID: "03922de4-1b33-4bd5-9a52-619a88b269b9"). InnerVolumeSpecName "kube-api-access-vtt78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:22:44.169745 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.169708    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtt78\" (UniqueName: \"kubernetes.io/projected/03922de4-1b33-4bd5-9a52-619a88b269b9-kube-api-access-vtt78\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:22:44.169745 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.169743    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:22:44.169745 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.169754    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03922de4-1b33-4bd5-9a52-619a88b269b9-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:22:44.820269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.820235    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57" event={"ID":"03922de4-1b33-4bd5-9a52-619a88b269b9","Type":"ContainerDied","Data":"fdad843ea5070cd1fbaee31f9a424603a593ab49fb3b632f9a4cfa452f90d4f8"}
Nov 02 02:22:44.820269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.820260    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"
Nov 02 02:22:44.820500 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:44.820284    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdad843ea5070cd1fbaee31f9a424603a593ab49fb3b632f9a4cfa452f90d4f8"
Nov 02 02:22:55.534656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:22:55.534619    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:23:05.016345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.016313    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"]
Nov 02 02:23:05.016841 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.016826    4912 cpu_manager.go:414] "containerMap: removing container" podUID="03922de4-1b33-4bd5-9a52-619a88b269b9" containerName="revoke-keys"
Nov 02 02:23:05.016894 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.016843    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03922de4-1b33-4bd5-9a52-619a88b269b9" containerName="revoke-keys"
Nov 02 02:23:05.016932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.016925    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="03922de4-1b33-4bd5-9a52-619a88b269b9" containerName="revoke-keys"
Nov 02 02:23:05.020547 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.020522    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.122656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.122626    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:23:05.122656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.122637    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:23:05.125833 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.125811    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"]
Nov 02 02:23:05.210820 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.210779    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wdf5w\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.210820 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.210822    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wdf5w\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.211050 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.210850    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4lb\" (UniqueName: \"kubernetes.io/projected/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-kube-api-access-fw4lb\") pod \"maas-api-revoke-sub-0b7e8edb-wdf5w\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.312639 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.312547    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wdf5w\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.312639 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.312583    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wdf5w\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.312639 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.312608    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4lb\" (UniqueName: \"kubernetes.io/projected/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-kube-api-access-fw4lb\") pod \"maas-api-revoke-sub-0b7e8edb-wdf5w\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.313324 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.313299    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wdf5w\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.315089 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.315045    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wdf5w\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.373201 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.373164    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4lb\" (UniqueName: \"kubernetes.io/projected/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-kube-api-access-fw4lb\") pod \"maas-api-revoke-sub-0b7e8edb-wdf5w\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:05.638772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:05.638737    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:06.014982 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:06.014953    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"]
Nov 02 02:23:06.015927 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:23:06.015899    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0c8876_6d0d_409b_b453_c2a2cfcc5a7b.slice/crio-19f3ea5b995000595b4c7f434b12fd7f5454eda820f36106921f682aab5ee923 WatchSource:0}: Error finding container 19f3ea5b995000595b4c7f434b12fd7f5454eda820f36106921f682aab5ee923: Status 404 returned error can't find the container with id 19f3ea5b995000595b4c7f434b12fd7f5454eda820f36106921f682aab5ee923
Nov 02 02:23:06.990804 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:06.990772    4912 generic.go:356] "Generic (PLEG): container finished" podID="6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b" containerID="3cb81adf4dc0e4331b95e8a2d97b0cc6e05228e28947dde31c814a71e2504aef" exitCode=0
Nov 02 02:23:06.991253 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:06.990866    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w" event={"ID":"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b","Type":"ContainerDied","Data":"3cb81adf4dc0e4331b95e8a2d97b0cc6e05228e28947dde31c814a71e2504aef"}
Nov 02 02:23:06.991253 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:06.990919    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w" event={"ID":"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b","Type":"ContainerStarted","Data":"19f3ea5b995000595b4c7f434b12fd7f5454eda820f36106921f682aab5ee923"}
Nov 02 02:23:08.138395 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.138368    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:08.243350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.243319    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-xks-ca\") pod \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") "
Nov 02 02:23:08.243570 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.243447    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-openshift-ca\") pod \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") "
Nov 02 02:23:08.243570 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.243510    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-kube-api-access-fw4lb\" (UniqueName: \"kubernetes.io/projected/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-kube-api-access-fw4lb\") pod \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\" (UID: \"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b\") "
Nov 02 02:23:08.243854 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.243825    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-openshift-ca" pod "6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b" (UID: "6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:23:08.245621 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.245596    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-xks-ca" pod "6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b" (UID: "6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:23:08.245763 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.245742    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-kube-api-access-fw4lb" pod "6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b" (UID: "6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b"). InnerVolumeSpecName "kube-api-access-fw4lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:23:08.345169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.345124    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4lb\" (UniqueName: \"kubernetes.io/projected/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-kube-api-access-fw4lb\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:08.345169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.345158    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:08.345169 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:08.345170    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:09.006911 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:09.006885    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"
Nov 02 02:23:09.006911 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:09.006892    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w" event={"ID":"6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b","Type":"ContainerDied","Data":"19f3ea5b995000595b4c7f434b12fd7f5454eda820f36106921f682aab5ee923"}
Nov 02 02:23:09.007195 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:09.006938    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f3ea5b995000595b4c7f434b12fd7f5454eda820f36106921f682aab5ee923"
Nov 02 02:23:12.138642 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:23:12.138593    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:23:16.361017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.360967    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"]
Nov 02 02:23:16.361685 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.361662    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b" containerName="revoke-keys"
Nov 02 02:23:16.361780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.361688    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b" containerName="revoke-keys"
Nov 02 02:23:16.361837 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.361804    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b" containerName="revoke-keys"
Nov 02 02:23:16.365122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.365099    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.419894 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.419853    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t478v\" (UniqueName: \"kubernetes.io/projected/68eba1ff-dc77-4e6f-a1d9-267a433ac912-kube-api-access-t478v\") pod \"maas-api-revoke-sub-3d4156f1-nq2qv\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.420121 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.419906    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nq2qv\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.420121 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.420038    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nq2qv\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.467345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.467308    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:23:16.467528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.467435    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:23:16.470578 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.470556    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"]
Nov 02 02:23:16.520669 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.520636    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t478v\" (UniqueName: \"kubernetes.io/projected/68eba1ff-dc77-4e6f-a1d9-267a433ac912-kube-api-access-t478v\") pod \"maas-api-revoke-sub-3d4156f1-nq2qv\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.520669 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.520670    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nq2qv\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.520893 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.520716    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nq2qv\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.521484 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.521462    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nq2qv\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.523356 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.523335    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nq2qv\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.583416 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.583379    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t478v\" (UniqueName: \"kubernetes.io/projected/68eba1ff-dc77-4e6f-a1d9-267a433ac912-kube-api-access-t478v\") pod \"maas-api-revoke-sub-3d4156f1-nq2qv\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:16.678914 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:16.678879    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:17.057823 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:17.057791    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"]
Nov 02 02:23:17.058985 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:23:17.058961    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68eba1ff_dc77_4e6f_a1d9_267a433ac912.slice/crio-8f7d6720a02ebae8ac7a4d70d81e69b9297b33bd691959cf076c9abbe475dd15 WatchSource:0}: Error finding container 8f7d6720a02ebae8ac7a4d70d81e69b9297b33bd691959cf076c9abbe475dd15: Status 404 returned error can't find the container with id 8f7d6720a02ebae8ac7a4d70d81e69b9297b33bd691959cf076c9abbe475dd15
Nov 02 02:23:18.066640 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:18.066601    4912 generic.go:356] "Generic (PLEG): container finished" podID="68eba1ff-dc77-4e6f-a1d9-267a433ac912" containerID="e21a0c4b9a6c33f704af24ca9519ff392b651bdc5da2f5a4a6a555ab27f4c14f" exitCode=0
Nov 02 02:23:18.067119 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:18.066690    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv" event={"ID":"68eba1ff-dc77-4e6f-a1d9-267a433ac912","Type":"ContainerDied","Data":"e21a0c4b9a6c33f704af24ca9519ff392b651bdc5da2f5a4a6a555ab27f4c14f"}
Nov 02 02:23:18.067119 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:18.066750    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv" event={"ID":"68eba1ff-dc77-4e6f-a1d9-267a433ac912","Type":"ContainerStarted","Data":"8f7d6720a02ebae8ac7a4d70d81e69b9297b33bd691959cf076c9abbe475dd15"}
Nov 02 02:23:19.536212 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.535842    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:19.591082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.585434    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-openshift-ca\") pod \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") "
Nov 02 02:23:19.591082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.585507    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-xks-ca\") pod \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") "
Nov 02 02:23:19.591082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.585633    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68eba1ff-dc77-4e6f-a1d9-267a433ac912-kube-api-access-t478v\" (UniqueName: \"kubernetes.io/projected/68eba1ff-dc77-4e6f-a1d9-267a433ac912-kube-api-access-t478v\") pod \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\" (UID: \"68eba1ff-dc77-4e6f-a1d9-267a433ac912\") "
Nov 02 02:23:19.619393 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.619314    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-openshift-ca" pod "68eba1ff-dc77-4e6f-a1d9-267a433ac912" (UID: "68eba1ff-dc77-4e6f-a1d9-267a433ac912"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:23:19.635867 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.635827    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68eba1ff-dc77-4e6f-a1d9-267a433ac912-kube-api-access-t478v" pod "68eba1ff-dc77-4e6f-a1d9-267a433ac912" (UID: "68eba1ff-dc77-4e6f-a1d9-267a433ac912"). InnerVolumeSpecName "kube-api-access-t478v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:23:19.637082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.636843    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-xks-ca" pod "68eba1ff-dc77-4e6f-a1d9-267a433ac912" (UID: "68eba1ff-dc77-4e6f-a1d9-267a433ac912"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:23:19.718848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.718754    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t478v\" (UniqueName: \"kubernetes.io/projected/68eba1ff-dc77-4e6f-a1d9-267a433ac912-kube-api-access-t478v\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:19.724091 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.719102    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:19.724295 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:19.724272    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68eba1ff-dc77-4e6f-a1d9-267a433ac912-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:20.109836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:20.109744    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv" event={"ID":"68eba1ff-dc77-4e6f-a1d9-267a433ac912","Type":"ContainerDied","Data":"8f7d6720a02ebae8ac7a4d70d81e69b9297b33bd691959cf076c9abbe475dd15"}
Nov 02 02:23:20.109836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:20.109778    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"
Nov 02 02:23:20.109836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:20.109822    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7d6720a02ebae8ac7a4d70d81e69b9297b33bd691959cf076c9abbe475dd15"
Nov 02 02:23:26.048198 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:26.048162    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"]
Nov 02 02:23:26.099731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:26.099689    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wrcqd"]
Nov 02 02:23:26.118903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:26.118871    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:23:27.229565 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:27.229467    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c39264-71a2-4166-b6e8-d6e0db7eb591" path="/var/lib/kubelet/pods/79c39264-71a2-4166-b6e8-d6e0db7eb591/volumes"
Nov 02 02:23:42.137169 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:23:42.137130    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:23:49.657176 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:49.657139    4912 scope.go:122] "RemoveContainer" containerID="bd0317818548c10263a8d8bfb76ad5d7aee0621f09486849e67440a95421707b"
Nov 02 02:23:50.031804 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.031715    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"]
Nov 02 02:23:50.081316 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.081282    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nq2qv"]
Nov 02 02:23:50.155345 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.155310    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"]
Nov 02 02:23:50.155838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.155822    4912 cpu_manager.go:414] "containerMap: removing container" podUID="68eba1ff-dc77-4e6f-a1d9-267a433ac912" containerName="revoke-keys"
Nov 02 02:23:50.155889 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.155840    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68eba1ff-dc77-4e6f-a1d9-267a433ac912" containerName="revoke-keys"
Nov 02 02:23:50.155948 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.155940    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="68eba1ff-dc77-4e6f-a1d9-267a433ac912" containerName="revoke-keys"
Nov 02 02:23:50.160584 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.160565    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.213200 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.213036    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:23:50.213463 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.213441    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:23:50.216836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.216810    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"]
Nov 02 02:23:50.317036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.316952    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfw5w\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.317036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.317016    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfw5w\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.317240 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.317097    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqfn\" (UniqueName: \"kubernetes.io/projected/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-kube-api-access-bwqfn\") pod \"maas-api-revoke-sub-3d4156f1-rfw5w\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.417866 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.417832    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfw5w\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.418085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.417902    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfw5w\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.418085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.417938    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqfn\" (UniqueName: \"kubernetes.io/projected/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-kube-api-access-bwqfn\") pod \"maas-api-revoke-sub-3d4156f1-rfw5w\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.418554 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.418533    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfw5w\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.420400 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.420381    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfw5w\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.479582 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.479549    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqfn\" (UniqueName: \"kubernetes.io/projected/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-kube-api-access-bwqfn\") pod \"maas-api-revoke-sub-3d4156f1-rfw5w\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:50.773943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:50.773912    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:51.151988 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:51.151959    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"]
Nov 02 02:23:51.152776 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:23:51.152746    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dadb97a_8152_4bfe_ab01_4688d9c4aa5e.slice/crio-622ee54a334212fae6d5c916cff9fc3895ef50416668ee969ed0543a1ffe3606 WatchSource:0}: Error finding container 622ee54a334212fae6d5c916cff9fc3895ef50416668ee969ed0543a1ffe3606: Status 404 returned error can't find the container with id 622ee54a334212fae6d5c916cff9fc3895ef50416668ee969ed0543a1ffe3606
Nov 02 02:23:51.224489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:51.224460    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68eba1ff-dc77-4e6f-a1d9-267a433ac912" path="/var/lib/kubelet/pods/68eba1ff-dc77-4e6f-a1d9-267a433ac912/volumes"
Nov 02 02:23:51.358734 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:51.358698    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w" event={"ID":"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e","Type":"ContainerStarted","Data":"622ee54a334212fae6d5c916cff9fc3895ef50416668ee969ed0543a1ffe3606"}
Nov 02 02:23:52.366703 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:52.366674    4912 generic.go:356] "Generic (PLEG): container finished" podID="9dadb97a-8152-4bfe-ab01-4688d9c4aa5e" containerID="f022b9f65e2a50caca76ab1cbd00d5296bc1bb62d00524e7b2dfb018aa4cf61e" exitCode=0
Nov 02 02:23:52.367193 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:52.366764    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w" event={"ID":"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e","Type":"ContainerDied","Data":"f022b9f65e2a50caca76ab1cbd00d5296bc1bb62d00524e7b2dfb018aa4cf61e"}
Nov 02 02:23:53.512352 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.512324    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:53.648106 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.648010    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-openshift-ca\") pod \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") "
Nov 02 02:23:53.648296 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.648175    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-xks-ca\") pod \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") "
Nov 02 02:23:53.648296 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.648233    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-kube-api-access-bwqfn\" (UniqueName: \"kubernetes.io/projected/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-kube-api-access-bwqfn\") pod \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\" (UID: \"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e\") "
Nov 02 02:23:53.648446 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.648349    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-openshift-ca" pod "9dadb97a-8152-4bfe-ab01-4688d9c4aa5e" (UID: "9dadb97a-8152-4bfe-ab01-4688d9c4aa5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:23:53.648515 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.648495    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:53.650546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.650513    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-xks-ca" pod "9dadb97a-8152-4bfe-ab01-4688d9c4aa5e" (UID: "9dadb97a-8152-4bfe-ab01-4688d9c4aa5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:23:53.650546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.650518    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-kube-api-access-bwqfn" pod "9dadb97a-8152-4bfe-ab01-4688d9c4aa5e" (UID: "9dadb97a-8152-4bfe-ab01-4688d9c4aa5e"). InnerVolumeSpecName "kube-api-access-bwqfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:23:53.749770 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.749736    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:53.749770 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:53.749765    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwqfn\" (UniqueName: \"kubernetes.io/projected/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e-kube-api-access-bwqfn\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:23:54.381203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:54.381178    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"
Nov 02 02:23:54.381386 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:54.381222    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w" event={"ID":"9dadb97a-8152-4bfe-ab01-4688d9c4aa5e","Type":"ContainerDied","Data":"622ee54a334212fae6d5c916cff9fc3895ef50416668ee969ed0543a1ffe3606"}
Nov 02 02:23:54.381386 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:54.381282    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="622ee54a334212fae6d5c916cff9fc3895ef50416668ee969ed0543a1ffe3606"
Nov 02 02:23:55.869264 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:23:55.869226    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:24:12.138827 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:24:12.138786    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:24:34.418085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:34.417185    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4tqpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:24:34.418085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:34.417277    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podUID="ea71fe46-235c-43bb-adc6-708430a55abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:24:35.058391 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:35.058352    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"]
Nov 02 02:24:35.107662 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:35.107626    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfdrj"]
Nov 02 02:24:35.224274 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:35.224240    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319b3f4a-962f-4eae-b2fa-3facce48ec17" path="/var/lib/kubelet/pods/319b3f4a-962f-4eae-b2fa-3facce48ec17/volumes"
Nov 02 02:24:40.381147 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.381022    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"]
Nov 02 02:24:40.381728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.381706    4912 cpu_manager.go:414] "containerMap: removing container" podUID="9dadb97a-8152-4bfe-ab01-4688d9c4aa5e" containerName="revoke-keys"
Nov 02 02:24:40.381819 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.381732    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dadb97a-8152-4bfe-ab01-4688d9c4aa5e" containerName="revoke-keys"
Nov 02 02:24:40.381907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.381893    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dadb97a-8152-4bfe-ab01-4688d9c4aa5e" containerName="revoke-keys"
Nov 02 02:24:40.385569 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.385543    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.443693 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.443660    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"]
Nov 02 02:24:40.488640 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.488613    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:24:40.488835 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.488774    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:24:40.516789 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.516752    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ph9vj\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.516960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.516872    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ph9vj\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.516960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.516936    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lr5q\" (UniqueName: \"kubernetes.io/projected/ec2d598d-7808-4939-9e61-d05be26346a2-kube-api-access-2lr5q\") pod \"maas-api-revoke-sub-e2fc9c5b-ph9vj\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.618516 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.618481    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lr5q\" (UniqueName: \"kubernetes.io/projected/ec2d598d-7808-4939-9e61-d05be26346a2-kube-api-access-2lr5q\") pod \"maas-api-revoke-sub-e2fc9c5b-ph9vj\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.618734 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.618541    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ph9vj\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.618734 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.618603    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ph9vj\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.619371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.619349    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ph9vj\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.621220 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.621200    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ph9vj\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.680090 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.679982    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lr5q\" (UniqueName: \"kubernetes.io/projected/ec2d598d-7808-4939-9e61-d05be26346a2-kube-api-access-2lr5q\") pod \"maas-api-revoke-sub-e2fc9c5b-ph9vj\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:40.698995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:40.698968    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:41.074553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:41.074524    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"]
Nov 02 02:24:41.075706 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:24:41.075679    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2d598d_7808_4939_9e61_d05be26346a2.slice/crio-40436a86f1590beecfd01358b5abe01d999bd2cf44b3ea7fbc85ae350baa42a5 WatchSource:0}: Error finding container 40436a86f1590beecfd01358b5abe01d999bd2cf44b3ea7fbc85ae350baa42a5: Status 404 returned error can't find the container with id 40436a86f1590beecfd01358b5abe01d999bd2cf44b3ea7fbc85ae350baa42a5
Nov 02 02:24:41.689389 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:41.689350    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj" event={"ID":"ec2d598d-7808-4939-9e61-d05be26346a2","Type":"ContainerStarted","Data":"2ca68200cb6e696c9b1bbba22bf68924776ce837dd09f038c5a5762b9bd6b7d8"}
Nov 02 02:24:41.689825 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:41.689415    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj" event={"ID":"ec2d598d-7808-4939-9e61-d05be26346a2","Type":"ContainerStarted","Data":"40436a86f1590beecfd01358b5abe01d999bd2cf44b3ea7fbc85ae350baa42a5"}
Nov 02 02:24:41.806656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:41.806596    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj" podStartSLOduration=1.8065787370000002 podStartE2EDuration="1.806578737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:24:41.804850309 +0000 UTC m=+3174.710315667" watchObservedRunningTime="2026-11-02 02:24:41.806578737 +0000 UTC m=+3174.712044094"
Nov 02 02:24:42.138523 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:24:42.138482    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:24:42.698264 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:42.698230    4912 generic.go:356] "Generic (PLEG): container finished" podID="ec2d598d-7808-4939-9e61-d05be26346a2" containerID="2ca68200cb6e696c9b1bbba22bf68924776ce837dd09f038c5a5762b9bd6b7d8" exitCode=0
Nov 02 02:24:42.698718 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:42.698273    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj" event={"ID":"ec2d598d-7808-4939-9e61-d05be26346a2","Type":"ContainerDied","Data":"2ca68200cb6e696c9b1bbba22bf68924776ce837dd09f038c5a5762b9bd6b7d8"}
Nov 02 02:24:43.851257 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:43.851224    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:43.952492 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:43.952460    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2d598d-7808-4939-9e61-d05be26346a2-kube-api-access-2lr5q\" (UniqueName: \"kubernetes.io/projected/ec2d598d-7808-4939-9e61-d05be26346a2-kube-api-access-2lr5q\") pod \"ec2d598d-7808-4939-9e61-d05be26346a2\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") "
Nov 02 02:24:43.952719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:43.952531    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-openshift-ca\") pod \"ec2d598d-7808-4939-9e61-d05be26346a2\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") "
Nov 02 02:24:43.952719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:43.952616    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-xks-ca\") pod \"ec2d598d-7808-4939-9e61-d05be26346a2\" (UID: \"ec2d598d-7808-4939-9e61-d05be26346a2\") "
Nov 02 02:24:43.952958 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:43.952934    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-openshift-ca" pod "ec2d598d-7808-4939-9e61-d05be26346a2" (UID: "ec2d598d-7808-4939-9e61-d05be26346a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:24:43.954950 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:43.954896    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-xks-ca" pod "ec2d598d-7808-4939-9e61-d05be26346a2" (UID: "ec2d598d-7808-4939-9e61-d05be26346a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:24:43.955092 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:43.955070    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2d598d-7808-4939-9e61-d05be26346a2-kube-api-access-2lr5q" pod "ec2d598d-7808-4939-9e61-d05be26346a2" (UID: "ec2d598d-7808-4939-9e61-d05be26346a2"). InnerVolumeSpecName "kube-api-access-2lr5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:44.054166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:44.054126    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lr5q\" (UniqueName: \"kubernetes.io/projected/ec2d598d-7808-4939-9e61-d05be26346a2-kube-api-access-2lr5q\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:44.054166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:44.054160    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:44.054166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:44.054171    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d598d-7808-4939-9e61-d05be26346a2-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:44.711803 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:44.711771    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"
Nov 02 02:24:44.712009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:44.711812    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj" event={"ID":"ec2d598d-7808-4939-9e61-d05be26346a2","Type":"ContainerDied","Data":"40436a86f1590beecfd01358b5abe01d999bd2cf44b3ea7fbc85ae350baa42a5"}
Nov 02 02:24:44.712009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:44.711871    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40436a86f1590beecfd01358b5abe01d999bd2cf44b3ea7fbc85ae350baa42a5"
Nov 02 02:24:49.690611 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:49.690577    4912 scope.go:122] "RemoveContainer" containerID="4dda39d3eb2d9d85cc1d71b2a954f690a39e67a06e77670dd8dd5631635a3ad8"
Nov 02 02:24:51.168794 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.168753    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"]
Nov 02 02:24:51.169566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.169544    4912 cpu_manager.go:414] "containerMap: removing container" podUID="ec2d598d-7808-4939-9e61-d05be26346a2" containerName="revoke-keys"
Nov 02 02:24:51.169620 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.169569    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2d598d-7808-4939-9e61-d05be26346a2" containerName="revoke-keys"
Nov 02 02:24:51.169754 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.169741    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2d598d-7808-4939-9e61-d05be26346a2" containerName="revoke-keys"
Nov 02 02:24:51.174664 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.174640    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.223505 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.223404    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-6d8rj\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.225237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.225183    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-6d8rj\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.225904 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.225881    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjsz2\" (UniqueName: \"kubernetes.io/projected/27bf4904-9555-40fb-9018-eec71e5e2e1b-kube-api-access-jjsz2\") pod \"maas-api-revoke-sub-5a29efb3-6d8rj\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.229367 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.229337    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"]
Nov 02 02:24:51.273455 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.273421    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:24:51.273632 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.273474    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:24:51.327306 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.327273    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-6d8rj\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.327306 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.327308    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjsz2\" (UniqueName: \"kubernetes.io/projected/27bf4904-9555-40fb-9018-eec71e5e2e1b-kube-api-access-jjsz2\") pod \"maas-api-revoke-sub-5a29efb3-6d8rj\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.327616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.327340    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-6d8rj\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.328017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.327992    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-6d8rj\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.329833 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.329812    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-6d8rj\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.389151 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.389112    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjsz2\" (UniqueName: \"kubernetes.io/projected/27bf4904-9555-40fb-9018-eec71e5e2e1b-kube-api-access-jjsz2\") pod \"maas-api-revoke-sub-5a29efb3-6d8rj\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.487811 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.487737    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:51.865425 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:51.865396    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"]
Nov 02 02:24:51.866133 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:24:51.866106    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bf4904_9555_40fb_9018_eec71e5e2e1b.slice/crio-b522a16a537dc0d16327b61a6fe6fc1c366ec6aa64d988d7ae116ddcc8d88b55 WatchSource:0}: Error finding container b522a16a537dc0d16327b61a6fe6fc1c366ec6aa64d988d7ae116ddcc8d88b55: Status 404 returned error can't find the container with id b522a16a537dc0d16327b61a6fe6fc1c366ec6aa64d988d7ae116ddcc8d88b55
Nov 02 02:24:52.764737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:52.764705    4912 generic.go:356] "Generic (PLEG): container finished" podID="27bf4904-9555-40fb-9018-eec71e5e2e1b" containerID="97f569ac271b11ace52f43e585e3978422ac6639f90fc0fe71c4364d8dc9cd7f" exitCode=0
Nov 02 02:24:52.765238 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:52.764793    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj" event={"ID":"27bf4904-9555-40fb-9018-eec71e5e2e1b","Type":"ContainerDied","Data":"97f569ac271b11ace52f43e585e3978422ac6639f90fc0fe71c4364d8dc9cd7f"}
Nov 02 02:24:52.765238 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:52.764863    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj" event={"ID":"27bf4904-9555-40fb-9018-eec71e5e2e1b","Type":"ContainerStarted","Data":"b522a16a537dc0d16327b61a6fe6fc1c366ec6aa64d988d7ae116ddcc8d88b55"}
Nov 02 02:24:53.952275 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:53.952236    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:54.054731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.054627    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27bf4904-9555-40fb-9018-eec71e5e2e1b-kube-api-access-jjsz2\" (UniqueName: \"kubernetes.io/projected/27bf4904-9555-40fb-9018-eec71e5e2e1b-kube-api-access-jjsz2\") pod \"27bf4904-9555-40fb-9018-eec71e5e2e1b\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") "
Nov 02 02:24:54.054931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.054798    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-openshift-ca\") pod \"27bf4904-9555-40fb-9018-eec71e5e2e1b\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") "
Nov 02 02:24:54.054931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.054882    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-xks-ca\") pod \"27bf4904-9555-40fb-9018-eec71e5e2e1b\" (UID: \"27bf4904-9555-40fb-9018-eec71e5e2e1b\") "
Nov 02 02:24:54.055360 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.055328    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-openshift-ca" pod "27bf4904-9555-40fb-9018-eec71e5e2e1b" (UID: "27bf4904-9555-40fb-9018-eec71e5e2e1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:24:54.057851 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.057814    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bf4904-9555-40fb-9018-eec71e5e2e1b-kube-api-access-jjsz2" pod "27bf4904-9555-40fb-9018-eec71e5e2e1b" (UID: "27bf4904-9555-40fb-9018-eec71e5e2e1b"). InnerVolumeSpecName "kube-api-access-jjsz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:54.058012 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.057873    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-xks-ca" pod "27bf4904-9555-40fb-9018-eec71e5e2e1b" (UID: "27bf4904-9555-40fb-9018-eec71e5e2e1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:24:54.156023 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.155985    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:54.156023 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.156020    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjsz2\" (UniqueName: \"kubernetes.io/projected/27bf4904-9555-40fb-9018-eec71e5e2e1b-kube-api-access-jjsz2\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:54.156361 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.156035    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bf4904-9555-40fb-9018-eec71e5e2e1b-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:54.819083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.818324    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj" event={"ID":"27bf4904-9555-40fb-9018-eec71e5e2e1b","Type":"ContainerDied","Data":"b522a16a537dc0d16327b61a6fe6fc1c366ec6aa64d988d7ae116ddcc8d88b55"}
Nov 02 02:24:54.819083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.818393    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b522a16a537dc0d16327b61a6fe6fc1c366ec6aa64d988d7ae116ddcc8d88b55"
Nov 02 02:24:54.819083 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:54.818565    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"
Nov 02 02:24:56.258903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:24:56.258866    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:06.056836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:06.056753    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"]
Nov 02 02:25:06.107352 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:06.107253    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xbnn"]
Nov 02 02:25:07.224896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:07.224851    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6877d44a-d9d4-4abd-ad99-754ea7ffe480" path="/var/lib/kubelet/pods/6877d44a-d9d4-4abd-ad99-754ea7ffe480/volumes"
Nov 02 02:25:08.082961 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.082924    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"]
Nov 02 02:25:08.132553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.132519    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfw5w"]
Nov 02 02:25:08.204153 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.204115    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"]
Nov 02 02:25:08.204610 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.204597    4912 cpu_manager.go:414] "containerMap: removing container" podUID="27bf4904-9555-40fb-9018-eec71e5e2e1b" containerName="revoke-keys"
Nov 02 02:25:08.204664 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.204612    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27bf4904-9555-40fb-9018-eec71e5e2e1b" containerName="revoke-keys"
Nov 02 02:25:08.204702 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.204687    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="27bf4904-9555-40fb-9018-eec71e5e2e1b" containerName="revoke-keys"
Nov 02 02:25:08.208160 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.208143    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.253530 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.253496    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkgs8\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.254009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.253548    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkgs8\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.254009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.253740    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndcnc\" (UniqueName: \"kubernetes.io/projected/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-kube-api-access-ndcnc\") pod \"maas-api-revoke-sub-3d4156f1-rkgs8\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.261836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.261800    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:25:08.262354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.262322    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:25:08.266997 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.266974    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"]
Nov 02 02:25:08.354781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.354698    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkgs8\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.354781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.354758    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkgs8\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.354975 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.354835    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndcnc\" (UniqueName: \"kubernetes.io/projected/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-kube-api-access-ndcnc\") pod \"maas-api-revoke-sub-3d4156f1-rkgs8\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.355632 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.355599    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkgs8\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.357354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.357324    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkgs8\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.415899 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.415867    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndcnc\" (UniqueName: \"kubernetes.io/projected/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-kube-api-access-ndcnc\") pod \"maas-api-revoke-sub-3d4156f1-rkgs8\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.520656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.520619    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:08.900205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.900157    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"]
Nov 02 02:25:08.900398 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:25:08.900374    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d53a2a_75cd_4c75_b157_1f079ee64ccd.slice/crio-a10a444f085b93c565302e7c2047247ce5df6991d7552767034f35eda91d159e WatchSource:0}: Error finding container a10a444f085b93c565302e7c2047247ce5df6991d7552767034f35eda91d159e: Status 404 returned error can't find the container with id a10a444f085b93c565302e7c2047247ce5df6991d7552767034f35eda91d159e
Nov 02 02:25:08.929600 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:08.929573    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8" event={"ID":"d3d53a2a-75cd-4c75-b157-1f079ee64ccd","Type":"ContainerStarted","Data":"a10a444f085b93c565302e7c2047247ce5df6991d7552767034f35eda91d159e"}
Nov 02 02:25:09.224393 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:09.224356    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dadb97a-8152-4bfe-ab01-4688d9c4aa5e" path="/var/lib/kubelet/pods/9dadb97a-8152-4bfe-ab01-4688d9c4aa5e/volumes"
Nov 02 02:25:09.937686 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:09.937654    4912 generic.go:356] "Generic (PLEG): container finished" podID="d3d53a2a-75cd-4c75-b157-1f079ee64ccd" containerID="809df5385b060cd15c574ef6d3073b6ae0999fe1b127853143de7dc3a1a96a04" exitCode=0
Nov 02 02:25:09.938176 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:09.937721    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8" event={"ID":"d3d53a2a-75cd-4c75-b157-1f079ee64ccd","Type":"ContainerDied","Data":"809df5385b060cd15c574ef6d3073b6ae0999fe1b127853143de7dc3a1a96a04"}
Nov 02 02:25:11.101907 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.101880    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:11.179382 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.179345    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-kube-api-access-ndcnc\" (UniqueName: \"kubernetes.io/projected/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-kube-api-access-ndcnc\") pod \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") "
Nov 02 02:25:11.179592 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.179498    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-openshift-ca\") pod \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") "
Nov 02 02:25:11.179685 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.179603    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-xks-ca\") pod \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\" (UID: \"d3d53a2a-75cd-4c75-b157-1f079ee64ccd\") "
Nov 02 02:25:11.179847 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.179819    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-openshift-ca" pod "d3d53a2a-75cd-4c75-b157-1f079ee64ccd" (UID: "d3d53a2a-75cd-4c75-b157-1f079ee64ccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:25:11.181677 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.181640    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-xks-ca" pod "d3d53a2a-75cd-4c75-b157-1f079ee64ccd" (UID: "d3d53a2a-75cd-4c75-b157-1f079ee64ccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:25:11.181798 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.181688    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-kube-api-access-ndcnc" pod "d3d53a2a-75cd-4c75-b157-1f079ee64ccd" (UID: "d3d53a2a-75cd-4c75-b157-1f079ee64ccd"). InnerVolumeSpecName "kube-api-access-ndcnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:25:11.281156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.281125    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:25:11.281156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.281153    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndcnc\" (UniqueName: \"kubernetes.io/projected/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-kube-api-access-ndcnc\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:25:11.281156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.281164    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d53a2a-75cd-4c75-b157-1f079ee64ccd-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:25:11.952867 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.952837    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"
Nov 02 02:25:11.953051 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.952875    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8" event={"ID":"d3d53a2a-75cd-4c75-b157-1f079ee64ccd","Type":"ContainerDied","Data":"a10a444f085b93c565302e7c2047247ce5df6991d7552767034f35eda91d159e"}
Nov 02 02:25:11.953051 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:11.952925    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a10a444f085b93c565302e7c2047247ce5df6991d7552767034f35eda91d159e"
Nov 02 02:25:12.137555 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:25:12.137509    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:25:19.048857 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:19.048815    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"]
Nov 02 02:25:19.098563 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:19.098521    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clndb"]
Nov 02 02:25:19.224435 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:19.224402    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045aa839-475e-4fd8-935a-e7dd48c0f8d8" path="/var/lib/kubelet/pods/045aa839-475e-4fd8-935a-e7dd48c0f8d8/volumes"
Nov 02 02:25:41.463777 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:41.463738    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:41.706858 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:41.706814    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:42.136923 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:25:42.136889    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:25:45.294070 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:45.294016    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:45.546756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:45.546662    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:48.533897 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:48.533862    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:48.713215 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:48.713171    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:48.918790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:48.918756    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"]
Nov 02 02:25:48.919275 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:48.919256    4912 cpu_manager.go:414] "containerMap: removing container" podUID="d3d53a2a-75cd-4c75-b157-1f079ee64ccd" containerName="revoke-keys"
Nov 02 02:25:48.919275 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:48.919271    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d53a2a-75cd-4c75-b157-1f079ee64ccd" containerName="revoke-keys"
Nov 02 02:25:48.919506 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:48.919352    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d53a2a-75cd-4c75-b157-1f079ee64ccd" containerName="revoke-keys"
Nov 02 02:25:48.922681 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:48.922663    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:25:48.981085 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:48.981017    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"]
Nov 02 02:25:49.026328 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.026286    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-054a89\""
Nov 02 02:25:49.026600 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.026343    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-33164f32-dockercfg-pf5tc\""
Nov 02 02:25:49.068176 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.068145    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"]
Nov 02 02:25:49.071963 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.071945    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:25:49.083394 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.083366    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388b6859-2be1-47a0-a541-8afe305fa38f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz\" (UID: \"388b6859-2be1-47a0-a541-8afe305fa38f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:25:49.083534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.083439    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxz4\" (UniqueName: \"kubernetes.io/projected/388b6859-2be1-47a0-a541-8afe305fa38f-kube-api-access-frxz4\") pod \"payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz\" (UID: \"388b6859-2be1-47a0-a541-8afe305fa38f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:25:49.129538 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.129510    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"]
Nov 02 02:25:49.184931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.184848    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388b6859-2be1-47a0-a541-8afe305fa38f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz\" (UID: \"388b6859-2be1-47a0-a541-8afe305fa38f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:25:49.184931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.184898    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxz4\" (UniqueName: \"kubernetes.io/projected/388b6859-2be1-47a0-a541-8afe305fa38f-kube-api-access-frxz4\") pod \"payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz\" (UID: \"388b6859-2be1-47a0-a541-8afe305fa38f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:25:49.184931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.184930    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2sv\" (UniqueName: \"kubernetes.io/projected/ec370d8d-f911-4316-891f-85d095a3af79-kube-api-access-lq2sv\") pod \"payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h\" (UID: \"ec370d8d-f911-4316-891f-85d095a3af79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:25:49.185211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.184947    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec370d8d-f911-4316-891f-85d095a3af79-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h\" (UID: \"ec370d8d-f911-4316-891f-85d095a3af79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:25:49.185566 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.185546    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388b6859-2be1-47a0-a541-8afe305fa38f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz\" (UID: \"388b6859-2be1-47a0-a541-8afe305fa38f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:25:49.246324 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.246296    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxz4\" (UniqueName: \"kubernetes.io/projected/388b6859-2be1-47a0-a541-8afe305fa38f-kube-api-access-frxz4\") pod \"payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz\" (UID: \"388b6859-2be1-47a0-a541-8afe305fa38f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:25:49.286535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.286493    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2sv\" (UniqueName: \"kubernetes.io/projected/ec370d8d-f911-4316-891f-85d095a3af79-kube-api-access-lq2sv\") pod \"payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h\" (UID: \"ec370d8d-f911-4316-891f-85d095a3af79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:25:49.286535 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.286535    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec370d8d-f911-4316-891f-85d095a3af79-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h\" (UID: \"ec370d8d-f911-4316-891f-85d095a3af79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:25:49.287328 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.287306    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec370d8d-f911-4316-891f-85d095a3af79-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h\" (UID: \"ec370d8d-f911-4316-891f-85d095a3af79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:25:49.347597 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.347559    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2sv\" (UniqueName: \"kubernetes.io/projected/ec370d8d-f911-4316-891f-85d095a3af79-kube-api-access-lq2sv\") pod \"payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h\" (UID: \"ec370d8d-f911-4316-891f-85d095a3af79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:25:49.385920 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.385879    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:25:49.537393 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.537358    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:25:49.720492 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.720413    4912 scope.go:122] "RemoveContainer" containerID="78a6b6ca947516700e81d23dbc98e294292bb5728d18a5ac569bccf38ccb8da1"
Nov 02 02:25:49.762721 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.762689    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"]
Nov 02 02:25:49.763004 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:25:49.762982    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec370d8d_f911_4316_891f_85d095a3af79.slice/crio-18da14903cb192c2cc946b4561b6d639c0baed7519f50ac462a7a30b60491908 WatchSource:0}: Error finding container 18da14903cb192c2cc946b4561b6d639c0baed7519f50ac462a7a30b60491908: Status 404 returned error can't find the container with id 18da14903cb192c2cc946b4561b6d639c0baed7519f50ac462a7a30b60491908
Nov 02 02:25:49.914949 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:25:49.914920    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388b6859_2be1_47a0_a541_8afe305fa38f.slice/crio-18ea538f095fede04e608fdc5c728ed4b92112204743d53074c54f417d0d4ff7 WatchSource:0}: Error finding container 18ea538f095fede04e608fdc5c728ed4b92112204743d53074c54f417d0d4ff7: Status 404 returned error can't find the container with id 18ea538f095fede04e608fdc5c728ed4b92112204743d53074c54f417d0d4ff7
Nov 02 02:25:49.915597 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:49.915482    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"]
Nov 02 02:25:50.210384 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:50.210337    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz" event={"ID":"388b6859-2be1-47a0-a541-8afe305fa38f","Type":"ContainerStarted","Data":"5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8"}
Nov 02 02:25:50.210589 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:50.210504    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:25:50.210589 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:50.210532    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz" event={"ID":"388b6859-2be1-47a0-a541-8afe305fa38f","Type":"ContainerStarted","Data":"18ea538f095fede04e608fdc5c728ed4b92112204743d53074c54f417d0d4ff7"}
Nov 02 02:25:50.212051 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:50.212028    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h" event={"ID":"ec370d8d-f911-4316-891f-85d095a3af79","Type":"ContainerStarted","Data":"79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d"}
Nov 02 02:25:50.212201 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:50.212087    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h" event={"ID":"ec370d8d-f911-4316-891f-85d095a3af79","Type":"ContainerStarted","Data":"18da14903cb192c2cc946b4561b6d639c0baed7519f50ac462a7a30b60491908"}
Nov 02 02:25:50.212267 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:50.212193    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:25:50.327803 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:50.327693    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz" podStartSLOduration=2.327680152 podStartE2EDuration="2.327680152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:25:50.324805903 +0000 UTC m=+3243.230271259" watchObservedRunningTime="2026-11-02 02:25:50.327680152 +0000 UTC m=+3243.233145508"
Nov 02 02:25:50.444079 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:50.444013    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h" podStartSLOduration=1.443995712 podStartE2EDuration="1.443995712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:25:50.439819911 +0000 UTC m=+3243.345285265" watchObservedRunningTime="2026-11-02 02:25:50.443995712 +0000 UTC m=+3243.349461098"
Nov 02 02:25:51.468009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.467975    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"]
Nov 02 02:25:51.471809 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.471792    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.576312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.576282    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-054a89-serving-cert\""
Nov 02 02:25:51.576719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.576701    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2n4c5\""
Nov 02 02:25:51.576812 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.576715    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 02:25:51.579911 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.579887    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"]
Nov 02 02:25:51.611872 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.611838    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.612027 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.611977    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6tc\" (UniqueName: \"kubernetes.io/projected/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-kube-api-access-jc6tc\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.612138 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.612094    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.612138 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.612122    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.701640 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.701605    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:51.712993 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.712951    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6tc\" (UniqueName: \"kubernetes.io/projected/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-kube-api-access-jc6tc\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.713347 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.713041    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.713633 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.713589    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.713770 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.713694    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.713930 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:25:51.713910    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-serving-cert: secret "maas-api-e2e-worker-single-main-054a89-serving-cert" not found
Nov 02 02:25:51.714019 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:25:51.713984    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls podName:df4ebd88-8d00-4bcc-aff0-eb93178f9ee4 nodeName:}" failed. No retries permitted until 2026-11-02 02:25:52.213963312 +0000 UTC m=+3245.119428652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls") pod "maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p" (UID: "df4ebd88-8d00-4bcc-aff0-eb93178f9ee4") : secret "maas-api-e2e-worker-single-main-054a89-serving-cert" not found
Nov 02 02:25:51.714741 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.714705    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.716327 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.716301    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.777766 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.777681    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6tc\" (UniqueName: \"kubernetes.io/projected/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-kube-api-access-jc6tc\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:51.884579 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:51.884537    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:52.060023 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:52.059935    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:52.218164 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:52.218116    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:52.220958 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:52.220892    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:52.385419 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:52.385389    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:52.762310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:52.762278    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"]
Nov 02 02:25:52.767369 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:52.767342    4912 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:25:53.234805 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:53.234765    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p" event={"ID":"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4","Type":"ContainerStarted","Data":"a49032a9d8d6140ef3d1fa0735f95d5541c0088d57c2cb40f9be3445e9d70b83"}
Nov 02 02:25:54.245191 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:54.245155    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p" event={"ID":"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4","Type":"ContainerStarted","Data":"b8b5894b972e46d34894986db89b180f06ed04966d92a7172a5a434f65290be6"}
Nov 02 02:25:54.245634 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:54.245340    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:25:54.364918 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:54.364867    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p" podStartSLOduration=2.412346236 podStartE2EDuration="3.364853578s" totalImagesPullingTime="952.507342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:25:54.362094964 +0000 UTC m=+3247.267560321" watchObservedRunningTime="2026-11-02 02:25:54.364853578 +0000 UTC m=+3247.270318943"
Nov 02 02:25:55.048326 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:55.048292    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"]
Nov 02 02:25:55.098094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:55.098045    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b55kj"]
Nov 02 02:25:55.225441 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:55.225400    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d6f675-a15c-49ab-a9a2-deb923b8e974" path="/var/lib/kubelet/pods/b7d6f675-a15c-49ab-a9a2-deb923b8e974/volumes"
Nov 02 02:25:56.564030 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:56.563985    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:25:56.797322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:25:56.797278    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:00.257832 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:00.257801    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:26:01.224737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:01.224709    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:26:01.224956 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:01.224783    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:26:01.682690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:01.682630    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:05.061375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:05.061332    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"]
Nov 02 02:26:05.109739 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:05.109700    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vn9h2"]
Nov 02 02:26:05.225957 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:05.225917    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcbeda3e-c4bf-420d-973a-229374d6658b" path="/var/lib/kubelet/pods/fcbeda3e-c4bf-420d-973a-229374d6658b/volumes"
Nov 02 02:26:06.579567 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:06.579517    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:10.071931 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:10.071892    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:10.680855 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:10.680821    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:10.861615 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:10.861580    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:11.770486 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:11.770447    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:12.137610 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:26:12.137576    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:26:13.430963 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:13.430928    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:15.382161 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:15.382124    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:16.685310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:16.685266    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:21.551287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:21.551249    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:26.356109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:26.356045    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:31.169003 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:31.168916    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:35.944456 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:35.944420    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:40.770363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:40.770323    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:42.138848 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:26:42.138811    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:26:43.049810 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:43.049773    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"]
Nov 02 02:26:43.099386 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:43.099344    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-78n7t"]
Nov 02 02:26:43.231422 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:43.231388    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368" path="/var/lib/kubelet/pods/62cc28ce-af4c-4c6f-9cf5-3ad94cb3d368/volumes"
Nov 02 02:26:45.583522 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:45.583306    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:47.930254 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:47.930226    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:26:47.936799 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:47.936776    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:26:48.080355 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:48.080323    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:26:48.088873 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:48.088846    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:26:49.746195 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:49.746161    4912 scope.go:122] "RemoveContainer" containerID="10180aca3230887c34db14e142707bf6dbd4ba4349614adb0bbfd4fc4856ae09"
Nov 02 02:26:49.758178 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:49.758154    4912 scope.go:122] "RemoveContainer" containerID="d3d60ae9d084e1610207e98c80ac33acd79758faf63fc474d8841190f636e23d"
Nov 02 02:26:50.460020 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:50.459986    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:55.244398 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:55.244360    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:26:56.956015 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:26:56.955978    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:00.068307 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:00.068245    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:04.857546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:04.857508    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:09.757612 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:09.757567    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:10.027892 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:10.027813    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:12.139264 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:27:12.139226    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:27:14.091110 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:14.091074    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"]
Nov 02 02:27:14.138472 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:14.138444    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lp7xl"]
Nov 02 02:27:14.916647 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:14.916614    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:15.125450 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:15.125410    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:15.225288 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:15.225208    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16d22bc-876f-48b8-8fc3-5d4d133f354d" path="/var/lib/kubelet/pods/e16d22bc-876f-48b8-8fc3-5d4d133f354d/volumes"
Nov 02 02:27:19.839118 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:19.839079    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:22.015461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:22.015424    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"]
Nov 02 02:27:22.124409 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:22.124376    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"]
Nov 02 02:27:22.412427 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:22.412382    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:22.641274 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:22.641231    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:24.547783 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:24.547734    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:27.016454 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:27.016394    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h" podUID="ec370d8d-f911-4316-891f-85d095a3af79" containerName="payload-processing" containerID="cri-o://79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d" gracePeriod=25
Nov 02 02:27:27.124934 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:27.124874    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz" podUID="388b6859-2be1-47a0-a541-8afe305fa38f" containerName="payload-pre-processing" containerID="cri-o://5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8" gracePeriod=25
Nov 02 02:27:28.635870 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:28.635831    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"]
Nov 02 02:27:28.636372 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:28.636030    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p" podUID="df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" containerName="maas-api" containerID="cri-o://b8b5894b972e46d34894986db89b180f06ed04966d92a7172a5a434f65290be6" gracePeriod=30
Nov 02 02:27:28.768090 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:28.768033    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:29.903580 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:29.903540    4912 generic.go:356] "Generic (PLEG): container finished" podID="df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" containerID="b8b5894b972e46d34894986db89b180f06ed04966d92a7172a5a434f65290be6" exitCode=0
Nov 02 02:27:29.903938 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:29.903610    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p" event={"ID":"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4","Type":"ContainerDied","Data":"b8b5894b972e46d34894986db89b180f06ed04966d92a7172a5a434f65290be6"}
Nov 02 02:27:29.903938 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:29.903695    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p" event={"ID":"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4","Type":"ContainerDied","Data":"a49032a9d8d6140ef3d1fa0735f95d5541c0088d57c2cb40f9be3445e9d70b83"}
Nov 02 02:27:29.903938 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:29.903715    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a49032a9d8d6140ef3d1fa0735f95d5541c0088d57c2cb40f9be3445e9d70b83"
Nov 02 02:27:29.904951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:29.904934    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:27:30.035708 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.035626    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls\") pod \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") "
Nov 02 02:27:30.035906 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.035723    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-metrics-certs\") pod \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") "
Nov 02 02:27:30.035906 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.035869    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-kube-api-access-jc6tc\" (UniqueName: \"kubernetes.io/projected/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-kube-api-access-jc6tc\") pod \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") "
Nov 02 02:27:30.036617 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.035901    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-openshift-service-ca\") pod \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\" (UID: \"df4ebd88-8d00-4bcc-aff0-eb93178f9ee4\") "
Nov 02 02:27:30.036617 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.036277    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-openshift-service-ca" pod "df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" (UID: "df4ebd88-8d00-4bcc-aff0-eb93178f9ee4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:27:30.036857 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.036828    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:30.038807 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.038764    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-kube-api-access-jc6tc" pod "df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" (UID: "df4ebd88-8d00-4bcc-aff0-eb93178f9ee4"). InnerVolumeSpecName "kube-api-access-jc6tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:27:30.039200 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.039173    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-metrics-certs" pod "df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" (UID: "df4ebd88-8d00-4bcc-aff0-eb93178f9ee4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:27:30.039319 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.039253    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls" pod "df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" (UID: "df4ebd88-8d00-4bcc-aff0-eb93178f9ee4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:27:30.138411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.138374    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc6tc\" (UniqueName: \"kubernetes.io/projected/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-kube-api-access-jc6tc\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:30.138411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.138405    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:30.138411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.138415    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:30.318825 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.318738    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:30.601207 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.601114    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:30.910435 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:30.910400    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"
Nov 02 02:27:31.075237 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:31.075194    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"]
Nov 02 02:27:31.124356 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:31.124318    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-054a89-657c64cbbf-rqw8p"]
Nov 02 02:27:31.221337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:31.221226    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz" podUID="388b6859-2be1-47a0-a541-8afe305fa38f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:27:31.221337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:31.221277    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h" podUID="ec370d8d-f911-4316-891f-85d095a3af79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:27:31.228434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:31.228400    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" path="/var/lib/kubelet/pods/df4ebd88-8d00-4bcc-aff0-eb93178f9ee4/volumes"
Nov 02 02:27:31.364622 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:31.364583    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:37.266493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:37.266442    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:37.451311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:37.451277    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:37.636875 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:37.636844    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:39.085553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.085520    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"]
Nov 02 02:27:39.135095 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.135051    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkgs8"]
Nov 02 02:27:39.206608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.206575    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"]
Nov 02 02:27:39.207089 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.207076    4912 cpu_manager.go:414] "containerMap: removing container" podUID="df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" containerName="maas-api"
Nov 02 02:27:39.207141 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.207091    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" containerName="maas-api"
Nov 02 02:27:39.207179 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.207169    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4ebd88-8d00-4bcc-aff0-eb93178f9ee4" containerName="maas-api"
Nov 02 02:27:39.211559 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.211541    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.224734 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.224708    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d53a2a-75cd-4c75-b157-1f079ee64ccd" path="/var/lib/kubelet/pods/d3d53a2a-75cd-4c75-b157-1f079ee64ccd/volumes"
Nov 02 02:27:39.264383 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.264354    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:27:39.264508 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.264440    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:27:39.268432 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.268407    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"]
Nov 02 02:27:39.338003 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.337930    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsdbt\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.338003 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.337968    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsdbt\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.338274 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.338091    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc9hb\" (UniqueName: \"kubernetes.io/projected/2f607c38-aea4-4d18-8605-1159c379df08-kube-api-access-dc9hb\") pod \"maas-api-revoke-sub-3d4156f1-dsdbt\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.439306 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.439271    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsdbt\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.439516 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.439321    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsdbt\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.439516 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.439395    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc9hb\" (UniqueName: \"kubernetes.io/projected/2f607c38-aea4-4d18-8605-1159c379df08-kube-api-access-dc9hb\") pod \"maas-api-revoke-sub-3d4156f1-dsdbt\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.439913 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.439890    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsdbt\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.441918 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.441889    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsdbt\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.499886 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.499855    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc9hb\" (UniqueName: \"kubernetes.io/projected/2f607c38-aea4-4d18-8605-1159c379df08-kube-api-access-dc9hb\") pod \"maas-api-revoke-sub-3d4156f1-dsdbt\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.524808 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.524786    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:39.904092 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.904041    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"]
Nov 02 02:27:39.905234 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:27:39.905209    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f607c38_aea4_4d18_8605_1159c379df08.slice/crio-fd3a7735629189c22d52a5cf648d8c6bb0f7644e748dadc6508c69d48c7a91be WatchSource:0}: Error finding container fd3a7735629189c22d52a5cf648d8c6bb0f7644e748dadc6508c69d48c7a91be: Status 404 returned error can't find the container with id fd3a7735629189c22d52a5cf648d8c6bb0f7644e748dadc6508c69d48c7a91be
Nov 02 02:27:39.969435 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:39.969404    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt" event={"ID":"2f607c38-aea4-4d18-8605-1159c379df08","Type":"ContainerStarted","Data":"fd3a7735629189c22d52a5cf648d8c6bb0f7644e748dadc6508c69d48c7a91be"}
Nov 02 02:27:40.977072 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:40.977031    4912 generic.go:356] "Generic (PLEG): container finished" podID="2f607c38-aea4-4d18-8605-1159c379df08" containerID="9cdb24f43d6a6d29cafbcad9923c343256d94eaf20ae700efde4fbab33b600b5" exitCode=0
Nov 02 02:27:40.977470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:40.977122    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt" event={"ID":"2f607c38-aea4-4d18-8605-1159c379df08","Type":"ContainerDied","Data":"9cdb24f43d6a6d29cafbcad9923c343256d94eaf20ae700efde4fbab33b600b5"}
Nov 02 02:27:41.221227 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:41.221149    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h" podUID="ec370d8d-f911-4316-891f-85d095a3af79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:27:41.221227 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:41.221168    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz" podUID="388b6859-2be1-47a0-a541-8afe305fa38f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:27:42.126030 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.125999    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:42.138623 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:27:42.138583    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:27:42.264548 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.264461    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-openshift-ca\") pod \"2f607c38-aea4-4d18-8605-1159c379df08\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") "
Nov 02 02:27:42.264739 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.264570    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-xks-ca\") pod \"2f607c38-aea4-4d18-8605-1159c379df08\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") "
Nov 02 02:27:42.264739 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.264613    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f607c38-aea4-4d18-8605-1159c379df08-kube-api-access-dc9hb\" (UniqueName: \"kubernetes.io/projected/2f607c38-aea4-4d18-8605-1159c379df08-kube-api-access-dc9hb\") pod \"2f607c38-aea4-4d18-8605-1159c379df08\" (UID: \"2f607c38-aea4-4d18-8605-1159c379df08\") "
Nov 02 02:27:42.264887 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.264860    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-openshift-ca" pod "2f607c38-aea4-4d18-8605-1159c379df08" (UID: "2f607c38-aea4-4d18-8605-1159c379df08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:27:42.265115 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.265090    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.266921 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.266892    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f607c38-aea4-4d18-8605-1159c379df08-kube-api-access-dc9hb" pod "2f607c38-aea4-4d18-8605-1159c379df08" (UID: "2f607c38-aea4-4d18-8605-1159c379df08"). InnerVolumeSpecName "kube-api-access-dc9hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:27:42.267214 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.267187    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-xks-ca" pod "2f607c38-aea4-4d18-8605-1159c379df08" (UID: "2f607c38-aea4-4d18-8605-1159c379df08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:27:42.366638 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.366598    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f607c38-aea4-4d18-8605-1159c379df08-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.366638 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.366631    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc9hb\" (UniqueName: \"kubernetes.io/projected/2f607c38-aea4-4d18-8605-1159c379df08-kube-api-access-dc9hb\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.995163 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.995122    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt" event={"ID":"2f607c38-aea4-4d18-8605-1159c379df08","Type":"ContainerDied","Data":"fd3a7735629189c22d52a5cf648d8c6bb0f7644e748dadc6508c69d48c7a91be"}
Nov 02 02:27:42.995163 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.995149    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"
Nov 02 02:27:42.995403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:42.995187    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd3a7735629189c22d52a5cf648d8c6bb0f7644e748dadc6508c69d48c7a91be"
Nov 02 02:27:45.048039 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:45.047997    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"]
Nov 02 02:27:45.090429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:45.090392    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-86h57"]
Nov 02 02:27:45.224548 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:45.224519    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03922de4-1b33-4bd5-9a52-619a88b269b9" path="/var/lib/kubelet/pods/03922de4-1b33-4bd5-9a52-619a88b269b9/volumes"
Nov 02 02:27:47.276456 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.276389    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:27:47.379582 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.379556    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:27:47.419081 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.419036    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec370d8d-f911-4316-891f-85d095a3af79-kube-api-access-lq2sv\" (UniqueName: \"kubernetes.io/projected/ec370d8d-f911-4316-891f-85d095a3af79-kube-api-access-lq2sv\") pod \"ec370d8d-f911-4316-891f-85d095a3af79\" (UID: \"ec370d8d-f911-4316-891f-85d095a3af79\") "
Nov 02 02:27:47.419261 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.419148    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec370d8d-f911-4316-891f-85d095a3af79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec370d8d-f911-4316-891f-85d095a3af79-plugins-config-volume\") pod \"ec370d8d-f911-4316-891f-85d095a3af79\" (UID: \"ec370d8d-f911-4316-891f-85d095a3af79\") "
Nov 02 02:27:47.419543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.419517    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec370d8d-f911-4316-891f-85d095a3af79-plugins-config-volume" pod "ec370d8d-f911-4316-891f-85d095a3af79" (UID: "ec370d8d-f911-4316-891f-85d095a3af79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:27:47.421449 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.421423    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec370d8d-f911-4316-891f-85d095a3af79-kube-api-access-lq2sv" pod "ec370d8d-f911-4316-891f-85d095a3af79" (UID: "ec370d8d-f911-4316-891f-85d095a3af79"). InnerVolumeSpecName "kube-api-access-lq2sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:27:47.520363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.520329    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388b6859-2be1-47a0-a541-8afe305fa38f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388b6859-2be1-47a0-a541-8afe305fa38f-plugins-config-volume\") pod \"388b6859-2be1-47a0-a541-8afe305fa38f\" (UID: \"388b6859-2be1-47a0-a541-8afe305fa38f\") "
Nov 02 02:27:47.520559 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.520454    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388b6859-2be1-47a0-a541-8afe305fa38f-kube-api-access-frxz4\" (UniqueName: \"kubernetes.io/projected/388b6859-2be1-47a0-a541-8afe305fa38f-kube-api-access-frxz4\") pod \"388b6859-2be1-47a0-a541-8afe305fa38f\" (UID: \"388b6859-2be1-47a0-a541-8afe305fa38f\") "
Nov 02 02:27:47.520776 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.520751    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388b6859-2be1-47a0-a541-8afe305fa38f-plugins-config-volume" pod "388b6859-2be1-47a0-a541-8afe305fa38f" (UID: "388b6859-2be1-47a0-a541-8afe305fa38f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:27:47.520872 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.520856    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388b6859-2be1-47a0-a541-8afe305fa38f-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:47.520938 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.520873    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq2sv\" (UniqueName: \"kubernetes.io/projected/ec370d8d-f911-4316-891f-85d095a3af79-kube-api-access-lq2sv\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:47.520938 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.520883    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec370d8d-f911-4316-891f-85d095a3af79-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:47.522709 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.522682    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388b6859-2be1-47a0-a541-8afe305fa38f-kube-api-access-frxz4" pod "388b6859-2be1-47a0-a541-8afe305fa38f" (UID: "388b6859-2be1-47a0-a541-8afe305fa38f"). InnerVolumeSpecName "kube-api-access-frxz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:27:47.622407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:47.622374    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxz4\" (UniqueName: \"kubernetes.io/projected/388b6859-2be1-47a0-a541-8afe305fa38f-kube-api-access-frxz4\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:48.030309 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.030216    4912 generic.go:356] "Generic (PLEG): container finished" podID="ec370d8d-f911-4316-891f-85d095a3af79" containerID="79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d" exitCode=0
Nov 02 02:27:48.030309 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.030260    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"
Nov 02 02:27:48.030550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.030308    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h" event={"ID":"ec370d8d-f911-4316-891f-85d095a3af79","Type":"ContainerDied","Data":"79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d"}
Nov 02 02:27:48.030550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.030357    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h" event={"ID":"ec370d8d-f911-4316-891f-85d095a3af79","Type":"ContainerDied","Data":"18da14903cb192c2cc946b4561b6d639c0baed7519f50ac462a7a30b60491908"}
Nov 02 02:27:48.030550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.030382    4912 scope.go:122] "RemoveContainer" containerID="79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d"
Nov 02 02:27:48.032287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.032136    4912 generic.go:356] "Generic (PLEG): container finished" podID="388b6859-2be1-47a0-a541-8afe305fa38f" containerID="5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8" exitCode=0
Nov 02 02:27:48.032287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.032191    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz" event={"ID":"388b6859-2be1-47a0-a541-8afe305fa38f","Type":"ContainerDied","Data":"5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8"}
Nov 02 02:27:48.032287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.032253    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz" event={"ID":"388b6859-2be1-47a0-a541-8afe305fa38f","Type":"ContainerDied","Data":"18ea538f095fede04e608fdc5c728ed4b92112204743d53074c54f417d0d4ff7"}
Nov 02 02:27:48.032287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.032210    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"
Nov 02 02:27:48.043550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.043530    4912 scope.go:122] "RemoveContainer" containerID="79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d"
Nov 02 02:27:48.043808 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:27:48.043793    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d\": container with ID starting with 79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d not found: ID does not exist" containerID="79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d"
Nov 02 02:27:48.043850 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.043818    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d"} err="failed to get container status \"79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d\": rpc error: code = NotFound desc = could not find container \"79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d\": container with ID starting with 79b7211fa0dde16c1acaac02154fe6d0af49161d2150c2ff1ae6a01f0c674c6d not found: ID does not exist"
Nov 02 02:27:48.043850 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.043835    4912 scope.go:122] "RemoveContainer" containerID="5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8"
Nov 02 02:27:48.053038 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.053020    4912 scope.go:122] "RemoveContainer" containerID="5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8"
Nov 02 02:27:48.053345 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:27:48.053327    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8\": container with ID starting with 5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8 not found: ID does not exist" containerID="5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8"
Nov 02 02:27:48.053402 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.053354    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8"} err="failed to get container status \"5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8\": rpc error: code = NotFound desc = could not find container \"5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8\": container with ID starting with 5eae67f32451251eb3c1762ab831a0cddaba67e279d84d51f6470a4343dd80b8 not found: ID does not exist"
Nov 02 02:27:48.195264 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.195228    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"]
Nov 02 02:27:48.244707 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.244669    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-054a89-7c86558dd74b5h"]
Nov 02 02:27:48.353712 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.353633    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"]
Nov 02 02:27:48.358067 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:48.358025    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-054a89-7795dzx2tz"]
Nov 02 02:27:49.225010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:49.224973    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388b6859-2be1-47a0-a541-8afe305fa38f" path="/var/lib/kubelet/pods/388b6859-2be1-47a0-a541-8afe305fa38f/volumes"
Nov 02 02:27:49.225425 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:49.225409    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec370d8d-f911-4316-891f-85d095a3af79" path="/var/lib/kubelet/pods/ec370d8d-f911-4316-891f-85d095a3af79/volumes"
Nov 02 02:27:49.794438 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:49.794406    4912 scope.go:122] "RemoveContainer" containerID="a0dbc2369eefcda644d2efa1b6bca40aa1880ed7d4e51ec63bf9c67bc0af99d8"
Nov 02 02:27:49.804286 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:49.804262    4912 scope.go:122] "RemoveContainer" containerID="a0f7ff7d113ee9cc898a3e141dc27b2476c12b67dd2373e575b52483cb512d6d"
Nov 02 02:27:49.813439 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:49.813416    4912 scope.go:122] "RemoveContainer" containerID="a3738dfe481e3482fd5400b2fe7328993f11b1033c340bf95ba1b3756192c67b"
Nov 02 02:27:51.750918 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:51.750879    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:52.339710 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:52.339675    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:57.323363 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:57.323272    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:27:59.161613 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:27:59.161580    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:28:01.043568 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:01.043534    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"]
Nov 02 02:28:01.043995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:01.043725    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp" podUID="579e48ba-c588-4231-91ef-61c31ad241c4" containerName="manager" containerID="cri-o://af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df" gracePeriod=10
Nov 02 02:28:01.313133 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:01.313106    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 02:28:01.342128 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:01.342094    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579e48ba-c588-4231-91ef-61c31ad241c4-kube-api-access-kc5cb\" (UniqueName: \"kubernetes.io/projected/579e48ba-c588-4231-91ef-61c31ad241c4-kube-api-access-kc5cb\") pod \"579e48ba-c588-4231-91ef-61c31ad241c4\" (UID: \"579e48ba-c588-4231-91ef-61c31ad241c4\") "
Nov 02 02:28:01.342287 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:01.342250    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579e48ba-c588-4231-91ef-61c31ad241c4-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/579e48ba-c588-4231-91ef-61c31ad241c4-extensions-socket-volume\") pod \"579e48ba-c588-4231-91ef-61c31ad241c4\" (UID: \"579e48ba-c588-4231-91ef-61c31ad241c4\") "
Nov 02 02:28:01.342631 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:01.342608    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579e48ba-c588-4231-91ef-61c31ad241c4-extensions-socket-volume" pod "579e48ba-c588-4231-91ef-61c31ad241c4" (UID: "579e48ba-c588-4231-91ef-61c31ad241c4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:28:01.344401 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:01.344371    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579e48ba-c588-4231-91ef-61c31ad241c4-kube-api-access-kc5cb" pod "579e48ba-c588-4231-91ef-61c31ad241c4" (UID: "579e48ba-c588-4231-91ef-61c31ad241c4"). InnerVolumeSpecName "kube-api-access-kc5cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:28:01.443391 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:01.443354    4912 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/579e48ba-c588-4231-91ef-61c31ad241c4-extensions-socket-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:01.443391 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:01.443391    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5cb\" (UniqueName: \"kubernetes.io/projected/579e48ba-c588-4231-91ef-61c31ad241c4-kube-api-access-kc5cb\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:02.144307 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:02.144268    4912 generic.go:356] "Generic (PLEG): container finished" podID="579e48ba-c588-4231-91ef-61c31ad241c4" containerID="af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df" exitCode=0
Nov 02 02:28:02.144780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:02.144317    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"
Nov 02 02:28:02.144780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:02.144353    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp" event={"ID":"579e48ba-c588-4231-91ef-61c31ad241c4","Type":"ContainerDied","Data":"af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df"}
Nov 02 02:28:02.144780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:02.144408    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp" event={"ID":"579e48ba-c588-4231-91ef-61c31ad241c4","Type":"ContainerDied","Data":"befde5e5a80e883226a570057ad787c197eea4ac27c6dd0f5ee59aeeb265bf65"}
Nov 02 02:28:02.144780 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:02.144437    4912 scope.go:122] "RemoveContainer" containerID="af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df"
Nov 02 02:28:02.155913 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:02.155894    4912 scope.go:122] "RemoveContainer" containerID="af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df"
Nov 02 02:28:02.156237 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:28:02.156212    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df\": container with ID starting with af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df not found: ID does not exist" containerID="af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df"
Nov 02 02:28:02.156297 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:02.156252    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df"} err="failed to get container status \"af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df\": rpc error: code = NotFound desc = could not find container \"af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df\": container with ID starting with af05f03190b54e090c4894fce78e318f602811960ef82cce4a13fcb2813164df not found: ID does not exist"
Nov 02 02:28:02.359692 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:02.359662    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"]
Nov 02 02:28:02.409446 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:02.409359    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cvpvp"]
Nov 02 02:28:03.224336 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:03.224303    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="579e48ba-c588-4231-91ef-61c31ad241c4" path="/var/lib/kubelet/pods/579e48ba-c588-4231-91ef-61c31ad241c4/volumes"
Nov 02 02:28:09.055837 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:09.055799    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"]
Nov 02 02:28:09.105243 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:09.105210    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wdf5w"]
Nov 02 02:28:09.224677 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:09.224640    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b" path="/var/lib/kubelet/pods/6e0c8876-6d0d-409b-b453-c2a2cfcc5a7b/volumes"
Nov 02 02:28:12.137826 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:28:12.137780    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:28:42.139046 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:28:42.139003    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:28:49.857888 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:49.857857    4912 scope.go:122] "RemoveContainer" containerID="cd3cb112e79523f3d5d93a9f90b31385b3b55861a9840ac2e47b0cc5f75c52d4"
Nov 02 02:28:49.867322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:49.867301    4912 scope.go:122] "RemoveContainer" containerID="001cf04aed7dc66b617415420b2cb81b32a14d1ae144abbe0bd7f5087cbcb873"
Nov 02 02:28:57.683661 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:28:57.683618    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:29:04.618520 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.618487    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"]
Nov 02 02:29:04.618938 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.618925    4912 cpu_manager.go:414] "containerMap: removing container" podUID="2f607c38-aea4-4d18-8605-1159c379df08" containerName="revoke-keys"
Nov 02 02:29:04.618987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.618939    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f607c38-aea4-4d18-8605-1159c379df08" containerName="revoke-keys"
Nov 02 02:29:04.618987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.618956    4912 cpu_manager.go:414] "containerMap: removing container" podUID="579e48ba-c588-4231-91ef-61c31ad241c4" containerName="manager"
Nov 02 02:29:04.618987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.618961    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579e48ba-c588-4231-91ef-61c31ad241c4" containerName="manager"
Nov 02 02:29:04.618987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.618978    4912 cpu_manager.go:414] "containerMap: removing container" podUID="388b6859-2be1-47a0-a541-8afe305fa38f" containerName="payload-pre-processing"
Nov 02 02:29:04.618987 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.618984    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388b6859-2be1-47a0-a541-8afe305fa38f" containerName="payload-pre-processing"
Nov 02 02:29:04.619211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.618999    4912 cpu_manager.go:414] "containerMap: removing container" podUID="ec370d8d-f911-4316-891f-85d095a3af79" containerName="payload-processing"
Nov 02 02:29:04.619211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.619005    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec370d8d-f911-4316-891f-85d095a3af79" containerName="payload-processing"
Nov 02 02:29:04.619211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.619088    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="388b6859-2be1-47a0-a541-8afe305fa38f" containerName="payload-pre-processing"
Nov 02 02:29:04.619211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.619098    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec370d8d-f911-4316-891f-85d095a3af79" containerName="payload-processing"
Nov 02 02:29:04.619211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.619108    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f607c38-aea4-4d18-8605-1159c379df08" containerName="revoke-keys"
Nov 02 02:29:04.619211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.619115    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="579e48ba-c588-4231-91ef-61c31ad241c4" containerName="manager"
Nov 02 02:29:04.623168 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.623151    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:04.677357 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.677323    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"]
Nov 02 02:29:04.723235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.723208    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fbj5j\""
Nov 02 02:29:04.770304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.770264    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xffc\" (UniqueName: \"kubernetes.io/projected/fc76523a-ee72-4475-b966-2fbde9c0d0b5-kube-api-access-8xffc\") pod \"kuadrant-operator-controller-manager-678d87766f-dtvzz\" (UID: \"fc76523a-ee72-4475-b966-2fbde9c0d0b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:04.770448 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.770403    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc76523a-ee72-4475-b966-2fbde9c0d0b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dtvzz\" (UID: \"fc76523a-ee72-4475-b966-2fbde9c0d0b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:04.871176 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.871096    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xffc\" (UniqueName: \"kubernetes.io/projected/fc76523a-ee72-4475-b966-2fbde9c0d0b5-kube-api-access-8xffc\") pod \"kuadrant-operator-controller-manager-678d87766f-dtvzz\" (UID: \"fc76523a-ee72-4475-b966-2fbde9c0d0b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:04.871329 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.871200    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc76523a-ee72-4475-b966-2fbde9c0d0b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dtvzz\" (UID: \"fc76523a-ee72-4475-b966-2fbde9c0d0b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:04.871563 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.871544    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc76523a-ee72-4475-b966-2fbde9c0d0b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dtvzz\" (UID: \"fc76523a-ee72-4475-b966-2fbde9c0d0b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:04.934094 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.934048    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xffc\" (UniqueName: \"kubernetes.io/projected/fc76523a-ee72-4475-b966-2fbde9c0d0b5-kube-api-access-8xffc\") pod \"kuadrant-operator-controller-manager-678d87766f-dtvzz\" (UID: \"fc76523a-ee72-4475-b966-2fbde9c0d0b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:04.936925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:04.936899    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:05.365116 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:05.365092    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"]
Nov 02 02:29:05.367388 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:29:05.367356    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc76523a_ee72_4475_b966_2fbde9c0d0b5.slice/crio-9a482ec38034943f8713d656a9f596d9654d795f0e1e4b9b891f8db7f4d79d83 WatchSource:0}: Error finding container 9a482ec38034943f8713d656a9f596d9654d795f0e1e4b9b891f8db7f4d79d83: Status 404 returned error can't find the container with id 9a482ec38034943f8713d656a9f596d9654d795f0e1e4b9b891f8db7f4d79d83
Nov 02 02:29:05.531796 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:05.531765    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz" event={"ID":"fc76523a-ee72-4475-b966-2fbde9c0d0b5","Type":"ContainerStarted","Data":"deca37d9f363f780cab1c651a1641121d236bac4f833eb21ec227f81c1198c8f"}
Nov 02 02:29:05.532002 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:05.531813    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz" event={"ID":"fc76523a-ee72-4475-b966-2fbde9c0d0b5","Type":"ContainerStarted","Data":"9a482ec38034943f8713d656a9f596d9654d795f0e1e4b9b891f8db7f4d79d83"}
Nov 02 02:29:05.532002 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:05.531937    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:05.653467 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:05.653416    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz" podStartSLOduration=1.653404101 podStartE2EDuration="1.653404101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:29:05.650878166 +0000 UTC m=+3438.556343524" watchObservedRunningTime="2026-11-02 02:29:05.653404101 +0000 UTC m=+3438.558869458"
Nov 02 02:29:12.138086 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:29:12.138036    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:29:16.541268 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:16.541232    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dtvzz"
Nov 02 02:29:31.929762 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:31.929682    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"]
Nov 02 02:29:31.933353 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:31.933337    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:31.955975 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:31.955930    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bpgtg\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:31.956125 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:31.955997    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx47g\" (UniqueName: \"kubernetes.io/projected/954a54e4-91b9-499f-9c6b-e9311664f73f-kube-api-access-zx47g\") pod \"maas-api-revoke-sub-d576e68b-bpgtg\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:31.956218 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:31.956193    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bpgtg\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:32.036434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.036404    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:29:32.036619 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.036445    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:29:32.039672 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.039650    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"]
Nov 02 02:29:32.057422 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.057394    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bpgtg\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:32.057549 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.057437    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx47g\" (UniqueName: \"kubernetes.io/projected/954a54e4-91b9-499f-9c6b-e9311664f73f-kube-api-access-zx47g\") pod \"maas-api-revoke-sub-d576e68b-bpgtg\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:32.057549 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.057500    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bpgtg\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:32.058245 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.058223    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bpgtg\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:32.060030 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.060009    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bpgtg\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:32.118656 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.118620    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx47g\" (UniqueName: \"kubernetes.io/projected/954a54e4-91b9-499f-9c6b-e9311664f73f-kube-api-access-zx47g\") pod \"maas-api-revoke-sub-d576e68b-bpgtg\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:32.249863 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.249786    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:32.626708 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.626675    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"]
Nov 02 02:29:32.627789 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:29:32.627756    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954a54e4_91b9_499f_9c6b_e9311664f73f.slice/crio-4e5ff792f86dd42ad388492ea0cb6acd8e786387c0c6fd42768dd2a423d44d09 WatchSource:0}: Error finding container 4e5ff792f86dd42ad388492ea0cb6acd8e786387c0c6fd42768dd2a423d44d09: Status 404 returned error can't find the container with id 4e5ff792f86dd42ad388492ea0cb6acd8e786387c0c6fd42768dd2a423d44d09
Nov 02 02:29:32.738967 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:32.738933    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg" event={"ID":"954a54e4-91b9-499f-9c6b-e9311664f73f","Type":"ContainerStarted","Data":"4e5ff792f86dd42ad388492ea0cb6acd8e786387c0c6fd42768dd2a423d44d09"}
Nov 02 02:29:33.746036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:33.746002    4912 generic.go:356] "Generic (PLEG): container finished" podID="954a54e4-91b9-499f-9c6b-e9311664f73f" containerID="8231e99598752ce4c0c1a4401f139587c146bd19d1bf24bf817ff74e40e564bd" exitCode=0
Nov 02 02:29:33.746490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:33.746099    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg" event={"ID":"954a54e4-91b9-499f-9c6b-e9311664f73f","Type":"ContainerDied","Data":"8231e99598752ce4c0c1a4401f139587c146bd19d1bf24bf817ff74e40e564bd"}
Nov 02 02:29:34.897477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:34.897449    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:34.984153 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:34.984126    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-openshift-ca\") pod \"954a54e4-91b9-499f-9c6b-e9311664f73f\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") "
Nov 02 02:29:34.984329 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:34.984173    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954a54e4-91b9-499f-9c6b-e9311664f73f-kube-api-access-zx47g\" (UniqueName: \"kubernetes.io/projected/954a54e4-91b9-499f-9c6b-e9311664f73f-kube-api-access-zx47g\") pod \"954a54e4-91b9-499f-9c6b-e9311664f73f\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") "
Nov 02 02:29:34.984329 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:34.984229    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-xks-ca\") pod \"954a54e4-91b9-499f-9c6b-e9311664f73f\" (UID: \"954a54e4-91b9-499f-9c6b-e9311664f73f\") "
Nov 02 02:29:34.984490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:34.984468    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-openshift-ca" pod "954a54e4-91b9-499f-9c6b-e9311664f73f" (UID: "954a54e4-91b9-499f-9c6b-e9311664f73f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:29:34.984618 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:34.984599    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:34.986677 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:34.986651    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-xks-ca" pod "954a54e4-91b9-499f-9c6b-e9311664f73f" (UID: "954a54e4-91b9-499f-9c6b-e9311664f73f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:29:34.986762 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:34.986744    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954a54e4-91b9-499f-9c6b-e9311664f73f-kube-api-access-zx47g" pod "954a54e4-91b9-499f-9c6b-e9311664f73f" (UID: "954a54e4-91b9-499f-9c6b-e9311664f73f"). InnerVolumeSpecName "kube-api-access-zx47g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:29:35.085943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:35.085862    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx47g\" (UniqueName: \"kubernetes.io/projected/954a54e4-91b9-499f-9c6b-e9311664f73f-kube-api-access-zx47g\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:35.085943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:35.085892    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954a54e4-91b9-499f-9c6b-e9311664f73f-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:29:35.760963 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:35.760931    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"
Nov 02 02:29:35.760963 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:35.760954    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg" event={"ID":"954a54e4-91b9-499f-9c6b-e9311664f73f","Type":"ContainerDied","Data":"4e5ff792f86dd42ad388492ea0cb6acd8e786387c0c6fd42768dd2a423d44d09"}
Nov 02 02:29:35.761215 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:35.760999    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e5ff792f86dd42ad388492ea0cb6acd8e786387c0c6fd42768dd2a423d44d09"
Nov 02 02:29:42.136539 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:29:42.136500    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:29:44.048462 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:44.048426    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"]
Nov 02 02:29:44.098865 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:44.098825    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ph9vj"]
Nov 02 02:29:45.224800 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:45.224763    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2d598d-7808-4939-9e61-d05be26346a2" path="/var/lib/kubelet/pods/ec2d598d-7808-4939-9e61-d05be26346a2/volumes"
Nov 02 02:29:49.904266 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:49.904223    4912 scope.go:122] "RemoveContainer" containerID="e21a0c4b9a6c33f704af24ca9519ff392b651bdc5da2f5a4a6a555ab27f4c14f"
Nov 02 02:29:49.917591 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:49.917562    4912 scope.go:122] "RemoveContainer" containerID="3cb81adf4dc0e4331b95e8a2d97b0cc6e05228e28947dde31c814a71e2504aef"
Nov 02 02:29:55.046988 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:55.046950    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"]
Nov 02 02:29:55.096487 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:55.096457    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6d8rj"]
Nov 02 02:29:55.226847 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:55.226807    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27bf4904-9555-40fb-9018-eec71e5e2e1b" path="/var/lib/kubelet/pods/27bf4904-9555-40fb-9018-eec71e5e2e1b/volumes"
Nov 02 02:29:57.974726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:29:57.974686    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:00.166015 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.165969    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"]
Nov 02 02:30:00.166638 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.166616    4912 cpu_manager.go:414] "containerMap: removing container" podUID="954a54e4-91b9-499f-9c6b-e9311664f73f" containerName="revoke-keys"
Nov 02 02:30:00.166728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.166640    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954a54e4-91b9-499f-9c6b-e9311664f73f" containerName="revoke-keys"
Nov 02 02:30:00.166827 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.166814    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="954a54e4-91b9-499f-9c6b-e9311664f73f" containerName="revoke-keys"
Nov 02 02:30:00.170123 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.170102    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:00.228273 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.228240    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"]
Nov 02 02:30:00.274336 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.274309    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:30:00.278265 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.278238    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz25g\" (UniqueName: \"kubernetes.io/projected/068d700f-c4fb-42b3-82c1-f2ff6b614777-kube-api-access-zz25g\") pod \"maas-api-key-cleanup-29893110-44bpq\" (UID: \"068d700f-c4fb-42b3-82c1-f2ff6b614777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:00.278376 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.278312    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068d700f-c4fb-42b3-82c1-f2ff6b614777-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893110-44bpq\" (UID: \"068d700f-c4fb-42b3-82c1-f2ff6b614777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:00.379234 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.379201    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz25g\" (UniqueName: \"kubernetes.io/projected/068d700f-c4fb-42b3-82c1-f2ff6b614777-kube-api-access-zz25g\") pod \"maas-api-key-cleanup-29893110-44bpq\" (UID: \"068d700f-c4fb-42b3-82c1-f2ff6b614777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:00.379441 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.379298    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068d700f-c4fb-42b3-82c1-f2ff6b614777-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893110-44bpq\" (UID: \"068d700f-c4fb-42b3-82c1-f2ff6b614777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:00.380011 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.379986    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068d700f-c4fb-42b3-82c1-f2ff6b614777-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893110-44bpq\" (UID: \"068d700f-c4fb-42b3-82c1-f2ff6b614777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:00.440653 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.440585    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz25g\" (UniqueName: \"kubernetes.io/projected/068d700f-c4fb-42b3-82c1-f2ff6b614777-kube-api-access-zz25g\") pod \"maas-api-key-cleanup-29893110-44bpq\" (UID: \"068d700f-c4fb-42b3-82c1-f2ff6b614777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:00.483511 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.483475    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:00.858122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.858098    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"]
Nov 02 02:30:00.859908 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:30:00.859873    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068d700f_c4fb_42b3_82c1_f2ff6b614777.slice/crio-bf67b3628187458b8e16f991055e4dace93faa06a5b82a17969f155a1f103638 WatchSource:0}: Error finding container bf67b3628187458b8e16f991055e4dace93faa06a5b82a17969f155a1f103638: Status 404 returned error can't find the container with id bf67b3628187458b8e16f991055e4dace93faa06a5b82a17969f155a1f103638
Nov 02 02:30:00.924065 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:00.924020    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq" event={"ID":"068d700f-c4fb-42b3-82c1-f2ff6b614777","Type":"ContainerStarted","Data":"bf67b3628187458b8e16f991055e4dace93faa06a5b82a17969f155a1f103638"}
Nov 02 02:30:01.934659 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:01.934625    4912 generic.go:356] "Generic (PLEG): container finished" podID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerID="ce62f23ea5149e27692050ec21248c2d10cc6d43dc140d7723b204176356cba5" exitCode=60
Nov 02 02:30:01.935104 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:01.934666    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq" event={"ID":"068d700f-c4fb-42b3-82c1-f2ff6b614777","Type":"ContainerDied","Data":"ce62f23ea5149e27692050ec21248c2d10cc6d43dc140d7723b204176356cba5"}
Nov 02 02:30:01.935104 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:01.934943    4912 scope.go:122] "RemoveContainer" containerID="ce62f23ea5149e27692050ec21248c2d10cc6d43dc140d7723b204176356cba5"
Nov 02 02:30:02.943728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:02.943691    4912 generic.go:356] "Generic (PLEG): container finished" podID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerID="67df6f0653584b2b04ab26b05cc503c02dba08a0aea0996806bf8b518f7453d5" exitCode=60
Nov 02 02:30:02.944216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:02.943749    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq" event={"ID":"068d700f-c4fb-42b3-82c1-f2ff6b614777","Type":"ContainerDied","Data":"67df6f0653584b2b04ab26b05cc503c02dba08a0aea0996806bf8b518f7453d5"}
Nov 02 02:30:02.944216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:02.943807    4912 scope.go:122] "RemoveContainer" containerID="ce62f23ea5149e27692050ec21248c2d10cc6d43dc140d7723b204176356cba5"
Nov 02 02:30:02.944216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:02.944027    4912 scope.go:122] "RemoveContainer" containerID="67df6f0653584b2b04ab26b05cc503c02dba08a0aea0996806bf8b518f7453d5"
Nov 02 02:30:02.944399 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:30:02.944301    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893110-44bpq_odh-ai-gateway-infra(068d700f-c4fb-42b3-82c1-f2ff6b614777)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777"
Nov 02 02:30:03.952934 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:03.952892    4912 scope.go:122] "RemoveContainer" containerID="67df6f0653584b2b04ab26b05cc503c02dba08a0aea0996806bf8b518f7453d5"
Nov 02 02:30:03.953466 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:30:03.953239    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893110-44bpq_odh-ai-gateway-infra(068d700f-c4fb-42b3-82c1-f2ff6b614777)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777"
Nov 02 02:30:12.137470 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:30:12.137434    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:30:13.436692 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:13.436655    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:14.150897 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:14.150844    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:14.461224 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:14.461140    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:16.218012 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:16.217978    4912 scope.go:122] "RemoveContainer" containerID="67df6f0653584b2b04ab26b05cc503c02dba08a0aea0996806bf8b518f7453d5"
Nov 02 02:30:17.042721 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:17.042685    4912 generic.go:356] "Generic (PLEG): container finished" podID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerID="e91674f1f9acb5cbc7021be5dad21f52e49a78c3a940cf8a6823e1f7a33d8621" exitCode=60
Nov 02 02:30:17.042882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:17.042762    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq" event={"ID":"068d700f-c4fb-42b3-82c1-f2ff6b614777","Type":"ContainerDied","Data":"e91674f1f9acb5cbc7021be5dad21f52e49a78c3a940cf8a6823e1f7a33d8621"}
Nov 02 02:30:17.042882 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:17.042825    4912 scope.go:122] "RemoveContainer" containerID="67df6f0653584b2b04ab26b05cc503c02dba08a0aea0996806bf8b518f7453d5"
Nov 02 02:30:17.043205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:17.043185    4912 scope.go:122] "RemoveContainer" containerID="e91674f1f9acb5cbc7021be5dad21f52e49a78c3a940cf8a6823e1f7a33d8621"
Nov 02 02:30:17.043509 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:30:17.043478    4912 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893110-44bpq_odh-ai-gateway-infra(068d700f-c4fb-42b3-82c1-f2ff6b614777)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777"
Nov 02 02:30:18.167985 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:18.167951    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"]
Nov 02 02:30:18.310519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:18.310496    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:18.360510 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:18.360459    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068d700f-c4fb-42b3-82c1-f2ff6b614777-kube-api-access-zz25g\" (UniqueName: \"kubernetes.io/projected/068d700f-c4fb-42b3-82c1-f2ff6b614777-kube-api-access-zz25g\") pod \"068d700f-c4fb-42b3-82c1-f2ff6b614777\" (UID: \"068d700f-c4fb-42b3-82c1-f2ff6b614777\") "
Nov 02 02:30:18.360690 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:18.360537    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068d700f-c4fb-42b3-82c1-f2ff6b614777-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068d700f-c4fb-42b3-82c1-f2ff6b614777-maas-api-service-ca\") pod \"068d700f-c4fb-42b3-82c1-f2ff6b614777\" (UID: \"068d700f-c4fb-42b3-82c1-f2ff6b614777\") "
Nov 02 02:30:18.360910 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:18.360887    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068d700f-c4fb-42b3-82c1-f2ff6b614777-maas-api-service-ca" pod "068d700f-c4fb-42b3-82c1-f2ff6b614777" (UID: "068d700f-c4fb-42b3-82c1-f2ff6b614777"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:30:18.362849 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:18.362828    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068d700f-c4fb-42b3-82c1-f2ff6b614777-kube-api-access-zz25g" pod "068d700f-c4fb-42b3-82c1-f2ff6b614777" (UID: "068d700f-c4fb-42b3-82c1-f2ff6b614777"). InnerVolumeSpecName "kube-api-access-zz25g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:30:18.462066 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:18.461965    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz25g\" (UniqueName: \"kubernetes.io/projected/068d700f-c4fb-42b3-82c1-f2ff6b614777-kube-api-access-zz25g\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:18.462066 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:18.461995    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068d700f-c4fb-42b3-82c1-f2ff6b614777-maas-api-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:18.904144 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:18.904106    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:19.059002 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:19.058966    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"
Nov 02 02:30:19.059247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:19.058966    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq" event={"ID":"068d700f-c4fb-42b3-82c1-f2ff6b614777","Type":"ContainerDied","Data":"bf67b3628187458b8e16f991055e4dace93faa06a5b82a17969f155a1f103638"}
Nov 02 02:30:19.059247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:19.059197    4912 scope.go:122] "RemoveContainer" containerID="e91674f1f9acb5cbc7021be5dad21f52e49a78c3a940cf8a6823e1f7a33d8621"
Nov 02 02:30:19.084909 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:19.084878    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:19.231694 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:19.231614    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"]
Nov 02 02:30:19.270853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:19.270815    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:19.272262 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:19.272235    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893110-44bpq"]
Nov 02 02:30:21.224410 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:21.224378    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" path="/var/lib/kubelet/pods/068d700f-c4fb-42b3-82c1-f2ff6b614777/volumes"
Nov 02 02:30:22.155015 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.154974    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:22.383454 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.383417    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:22.542811 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.542730    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"]
Nov 02 02:30:22.543275 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.543260    4912 cpu_manager.go:414] "containerMap: removing container" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerName="cleanup"
Nov 02 02:30:22.543323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.543277    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerName="cleanup"
Nov 02 02:30:22.543323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.543291    4912 cpu_manager.go:414] "containerMap: removing container" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerName="cleanup"
Nov 02 02:30:22.543323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.543296    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerName="cleanup"
Nov 02 02:30:22.543429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.543383    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerName="cleanup"
Nov 02 02:30:22.543429 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.543396    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerName="cleanup"
Nov 02 02:30:22.546860 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.546841    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:22.605380 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.605348    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4532c7e-35a4-4b12-b258-cafabc53e5e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86\" (UID: \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:22.605560 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.605513    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x99hb\" (UniqueName: \"kubernetes.io/projected/c4532c7e-35a4-4b12-b258-cafabc53e5e7-kube-api-access-x99hb\") pod \"payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86\" (UID: \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:22.649978 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.649938    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-18df32\""
Nov 02 02:30:22.649978 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.649941    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-31e5f43b-dockercfg-47sfk\""
Nov 02 02:30:22.653299 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.653270    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"]
Nov 02 02:30:22.702408 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.702377    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"]
Nov 02 02:30:22.702839 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.702827    4912 cpu_manager.go:414] "containerMap: removing container" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerName="cleanup"
Nov 02 02:30:22.702897 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.702840    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerName="cleanup"
Nov 02 02:30:22.702940 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.702931    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="068d700f-c4fb-42b3-82c1-f2ff6b614777" containerName="cleanup"
Nov 02 02:30:22.706160 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.706134    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:22.706492 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.706466    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99hb\" (UniqueName: \"kubernetes.io/projected/c4532c7e-35a4-4b12-b258-cafabc53e5e7-kube-api-access-x99hb\") pod \"payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86\" (UID: \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:22.706580 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.706561    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4532c7e-35a4-4b12-b258-cafabc53e5e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86\" (UID: \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:22.707336 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.707316    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4532c7e-35a4-4b12-b258-cafabc53e5e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86\" (UID: \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:22.752583 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.752557    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"]
Nov 02 02:30:22.766871 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.766844    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99hb\" (UniqueName: \"kubernetes.io/projected/c4532c7e-35a4-4b12-b258-cafabc53e5e7-kube-api-access-x99hb\") pod \"payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86\" (UID: \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:22.807939 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.807874    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52647b8b-755e-4e68-b331-ad5c0419034d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2\" (UID: \"52647b8b-755e-4e68-b331-ad5c0419034d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:22.807939 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.807912    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbc92\" (UniqueName: \"kubernetes.io/projected/52647b8b-755e-4e68-b331-ad5c0419034d-kube-api-access-lbc92\") pod \"payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2\" (UID: \"52647b8b-755e-4e68-b331-ad5c0419034d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:22.860021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.859989    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:22.908640 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.908605    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52647b8b-755e-4e68-b331-ad5c0419034d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2\" (UID: \"52647b8b-755e-4e68-b331-ad5c0419034d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:22.908836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.908658    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbc92\" (UniqueName: \"kubernetes.io/projected/52647b8b-755e-4e68-b331-ad5c0419034d-kube-api-access-lbc92\") pod \"payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2\" (UID: \"52647b8b-755e-4e68-b331-ad5c0419034d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:22.910327 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.910272    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52647b8b-755e-4e68-b331-ad5c0419034d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2\" (UID: \"52647b8b-755e-4e68-b331-ad5c0419034d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:22.969425 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:22.969391    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbc92\" (UniqueName: \"kubernetes.io/projected/52647b8b-755e-4e68-b331-ad5c0419034d-kube-api-access-lbc92\") pod \"payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2\" (UID: \"52647b8b-755e-4e68-b331-ad5c0419034d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:23.020528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:23.020495    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:23.234639 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:23.234608    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"]
Nov 02 02:30:23.235922 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:30:23.235893    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4532c7e_35a4_4b12_b258_cafabc53e5e7.slice/crio-7eebd73cf700d50ac97da82cf080f334d6c998c3d14e6451d3e9f0ea2a88fa83 WatchSource:0}: Error finding container 7eebd73cf700d50ac97da82cf080f334d6c998c3d14e6451d3e9f0ea2a88fa83: Status 404 returned error can't find the container with id 7eebd73cf700d50ac97da82cf080f334d6c998c3d14e6451d3e9f0ea2a88fa83
Nov 02 02:30:23.399557 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:23.399510    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"]
Nov 02 02:30:23.401394 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:30:23.401364    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52647b8b_755e_4e68_b331_ad5c0419034d.slice/crio-2886fc00732790f7be4fce4ac62e3d0214607645766bf6e5b407f72d858d1981 WatchSource:0}: Error finding container 2886fc00732790f7be4fce4ac62e3d0214607645766bf6e5b407f72d858d1981: Status 404 returned error can't find the container with id 2886fc00732790f7be4fce4ac62e3d0214607645766bf6e5b407f72d858d1981
Nov 02 02:30:24.095820 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:24.095781    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86" event={"ID":"c4532c7e-35a4-4b12-b258-cafabc53e5e7","Type":"ContainerStarted","Data":"599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05"}
Nov 02 02:30:24.096024 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:24.095844    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86" event={"ID":"c4532c7e-35a4-4b12-b258-cafabc53e5e7","Type":"ContainerStarted","Data":"7eebd73cf700d50ac97da82cf080f334d6c998c3d14e6451d3e9f0ea2a88fa83"}
Nov 02 02:30:24.096024 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:24.095868    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:24.097526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:24.097497    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2" event={"ID":"52647b8b-755e-4e68-b331-ad5c0419034d","Type":"ContainerStarted","Data":"719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0"}
Nov 02 02:30:24.097671 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:24.097541    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2" event={"ID":"52647b8b-755e-4e68-b331-ad5c0419034d","Type":"ContainerStarted","Data":"2886fc00732790f7be4fce4ac62e3d0214607645766bf6e5b407f72d858d1981"}
Nov 02 02:30:24.097671 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:24.097578    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:24.212312 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:24.212249    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86" podStartSLOduration=2.212230254 podStartE2EDuration="2.212230254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:30:24.210192539 +0000 UTC m=+3517.115657897" watchObservedRunningTime="2026-11-02 02:30:24.212230254 +0000 UTC m=+3517.117695611"
Nov 02 02:30:24.326946 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:24.326883    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2" podStartSLOduration=2.326864254 podStartE2EDuration="2.326864254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:30:24.32462924 +0000 UTC m=+3517.230094597" watchObservedRunningTime="2026-11-02 02:30:24.326864254 +0000 UTC m=+3517.232329613"
Nov 02 02:30:25.055455 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.055423    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"]
Nov 02 02:30:25.059431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.059409    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.114023 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.113988    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-18df32-serving-cert\""
Nov 02 02:30:25.118009 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.117980    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"]
Nov 02 02:30:25.131543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.131507    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4j7\" (UniqueName: \"kubernetes.io/projected/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-kube-api-access-jz4j7\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.131708 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.131656    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.131788 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.131718    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.131877 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.131854    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.163592 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.163563    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 02:30:25.163752 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.163626    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2n4c5\""
Nov 02 02:30:25.232891 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.232862    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.233120 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.232917    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.233204 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.233122    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.233271 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.233233    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4j7\" (UniqueName: \"kubernetes.io/projected/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-kube-api-access-jz4j7\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.233271 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:30:25.233252    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-serving-cert: secret "maas-api-e2e-worker-single-main-18df32-serving-cert" not found
Nov 02 02:30:25.233389 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:30:25.233320    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls podName:9e08c1ee-66f2-4f91-974c-5dc7d710d35d nodeName:}" failed. No retries permitted until 2026-11-02 02:30:25.733299534 +0000 UTC m=+3518.638764875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls") pod "maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd" (UID: "9e08c1ee-66f2-4f91-974c-5dc7d710d35d") : secret "maas-api-e2e-worker-single-main-18df32-serving-cert" not found
Nov 02 02:30:25.233667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.233650    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.235580 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.235529    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.283728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.283695    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:25.297300 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.297229    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4j7\" (UniqueName: \"kubernetes.io/projected/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-kube-api-access-jz4j7\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.458937 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.458887    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:25.629834 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.629796    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:25.738395 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.738312    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.740964 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.740947    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:25.974235 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:25.974197    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:26.352856 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:26.352830    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"]
Nov 02 02:30:26.354983 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:30:26.354960    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e08c1ee_66f2_4f91_974c_5dc7d710d35d.slice/crio-5f94dfcad59906eefa716af4022e995d2639f748514ef85386158ca0b38d654b WatchSource:0}: Error finding container 5f94dfcad59906eefa716af4022e995d2639f748514ef85386158ca0b38d654b: Status 404 returned error can't find the container with id 5f94dfcad59906eefa716af4022e995d2639f748514ef85386158ca0b38d654b
Nov 02 02:30:27.121168 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:27.121123    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd" event={"ID":"9e08c1ee-66f2-4f91-974c-5dc7d710d35d","Type":"ContainerStarted","Data":"5f94dfcad59906eefa716af4022e995d2639f748514ef85386158ca0b38d654b"}
Nov 02 02:30:28.128760 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:28.128726    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd" event={"ID":"9e08c1ee-66f2-4f91-974c-5dc7d710d35d","Type":"ContainerStarted","Data":"5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd"}
Nov 02 02:30:28.129192 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:28.128818    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:28.246616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:28.246564    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd" podStartSLOduration=2.19926369 podStartE2EDuration="3.246545553s" totalImagesPullingTime="1.047281863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:30:28.245247901 +0000 UTC m=+3521.150713262" watchObservedRunningTime="2026-11-02 02:30:28.246545553 +0000 UTC m=+3521.152010911"
Nov 02 02:30:30.238491 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:30.238460    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:34.142032 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:34.142002    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:30:35.106867 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:35.106836    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:30:35.107082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:35.106905    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:30:35.183029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:35.182992    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:40.042538 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:40.042501    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:42.144825 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:30:42.144779    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:30:43.753023 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:43.752990    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:44.352870 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:44.352833    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:44.582324 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:44.582224    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:45.201922 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:45.201878    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:47.073206 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:47.073154    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:49.954584 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:49.954548    4912 scope.go:122] "RemoveContainer" containerID="f022b9f65e2a50caca76ab1cbd00d5296bc1bb62d00524e7b2dfb018aa4cf61e"
Nov 02 02:30:49.964489 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:49.964464    4912 scope.go:122] "RemoveContainer" containerID="2ca68200cb6e696c9b1bbba22bf68924776ce837dd09f038c5a5762b9bd6b7d8"
Nov 02 02:30:50.037531 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:50.037499    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:54.844715 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:54.844679    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:58.327032 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:58.326938    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:30:59.659403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:30:59.659369    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:03.996606 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:03.996577    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:04.539447 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:04.539393    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:09.371292 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:09.371256    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:12.138946 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:31:12.138905    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:31:14.184337 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:14.184298    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:18.962439 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:18.962404    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:23.853447 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:23.853406    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:28.654250 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:28.654210    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:33.462625 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:33.462586    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:34.760303 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:34.760267    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:34.941354 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:34.941314    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:38.253138 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:38.253102    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:42.138217 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:31:42.138176    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:31:43.070117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:43.070046    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:43.827001 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:43.826968    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:47.976040 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:47.976012    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:31:47.982586 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:47.982560    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:31:48.132365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:48.132333    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:31:48.139310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:48.139289    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:31:48.282398 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:48.282308    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:48.863691 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:48.863650    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:50.001311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:50.001276    4912 scope.go:122] "RemoveContainer" containerID="809df5385b060cd15c574ef6d3073b6ae0999fe1b127853143de7dc3a1a96a04"
Nov 02 02:31:50.011249 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:50.011226    4912 scope.go:122] "RemoveContainer" containerID="97f569ac271b11ace52f43e585e3978422ac6639f90fc0fe71c4364d8dc9cd7f"
Nov 02 02:31:53.251558 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:53.251522    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:55.759462 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:55.759428    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"]
Nov 02 02:31:55.863840 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:55.863804    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"]
Nov 02 02:31:56.146455 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:56.146405    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:56.374026 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:56.373991    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:58.051210 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:58.051166    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:31:58.672825 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:31:58.672793    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:00.760391 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:00.760351    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2" podUID="52647b8b-755e-4e68-b331-ad5c0419034d" containerName="payload-processing" containerID="cri-o://719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0" gracePeriod=25
Nov 02 02:32:00.864358 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:00.864315    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86" podUID="c4532c7e-35a4-4b12-b258-cafabc53e5e7" containerName="payload-pre-processing" containerID="cri-o://599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05" gracePeriod=25
Nov 02 02:32:02.131292 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:02.131256    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"]
Nov 02 02:32:02.131764 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:02.131443    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd" podUID="9e08c1ee-66f2-4f91-974c-5dc7d710d35d" containerName="maas-api" containerID="cri-o://5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd" gracePeriod=30
Nov 02 02:32:02.260450 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:02.260412    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:03.393575 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.393552    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:32:03.451093 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.450970    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls\") pod \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") "
Nov 02 02:32:03.451093 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.451039    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-kube-api-access-jz4j7\" (UniqueName: \"kubernetes.io/projected/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-kube-api-access-jz4j7\") pod \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") "
Nov 02 02:32:03.451340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.451124    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-metrics-certs\") pod \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") "
Nov 02 02:32:03.451340 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.451176    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-openshift-service-ca\") pod \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\" (UID: \"9e08c1ee-66f2-4f91-974c-5dc7d710d35d\") "
Nov 02 02:32:03.451618 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.451594    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-openshift-service-ca" pod "9e08c1ee-66f2-4f91-974c-5dc7d710d35d" (UID: "9e08c1ee-66f2-4f91-974c-5dc7d710d35d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:32:03.453231 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.453206    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls" pod "9e08c1ee-66f2-4f91-974c-5dc7d710d35d" (UID: "9e08c1ee-66f2-4f91-974c-5dc7d710d35d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:32:03.453392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.453370    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-kube-api-access-jz4j7" pod "9e08c1ee-66f2-4f91-974c-5dc7d710d35d" (UID: "9e08c1ee-66f2-4f91-974c-5dc7d710d35d"). InnerVolumeSpecName "kube-api-access-jz4j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:03.453438 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.453385    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-metrics-certs" pod "9e08c1ee-66f2-4f91-974c-5dc7d710d35d" (UID: "9e08c1ee-66f2-4f91-974c-5dc7d710d35d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:32:03.552017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.551981    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:03.552017 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.552016    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:03.552269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.552026    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4j7\" (UniqueName: \"kubernetes.io/projected/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-kube-api-access-jz4j7\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:03.552269 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.552037    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e08c1ee-66f2-4f91-974c-5dc7d710d35d-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:03.766688 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.766600    4912 generic.go:356] "Generic (PLEG): container finished" podID="9e08c1ee-66f2-4f91-974c-5dc7d710d35d" containerID="5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd" exitCode=0
Nov 02 02:32:03.766688 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.766652    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"
Nov 02 02:32:03.766934 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.766688    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd" event={"ID":"9e08c1ee-66f2-4f91-974c-5dc7d710d35d","Type":"ContainerDied","Data":"5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd"}
Nov 02 02:32:03.766934 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.766751    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd" event={"ID":"9e08c1ee-66f2-4f91-974c-5dc7d710d35d","Type":"ContainerDied","Data":"5f94dfcad59906eefa716af4022e995d2639f748514ef85386158ca0b38d654b"}
Nov 02 02:32:03.766934 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.766781    4912 scope.go:122] "RemoveContainer" containerID="5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd"
Nov 02 02:32:03.777050 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.777026    4912 scope.go:122] "RemoveContainer" containerID="5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd"
Nov 02 02:32:03.777355 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:32:03.777334    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd\": container with ID starting with 5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd not found: ID does not exist" containerID="5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd"
Nov 02 02:32:03.777398 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.777364    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd"} err="failed to get container status \"5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd\": rpc error: code = NotFound desc = could not find container \"5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd\": container with ID starting with 5e093968d7d8a649614c2802092edd5b4f786e40b905214f9c3ccfa088e0f5bd not found: ID does not exist"
Nov 02 02:32:03.810653 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.810622    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:03.932214 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.932184    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"]
Nov 02 02:32:03.937626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:03.937603    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-18df32-67988674b4-c8bpd"]
Nov 02 02:32:04.153722 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:04.153689    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:04.920308 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:04.920273    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:05.105178 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:05.105133    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86" podUID="c4532c7e-35a4-4b12-b258-cafabc53e5e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:32:05.105371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:05.105199    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2" podUID="52647b8b-755e-4e68-b331-ad5c0419034d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:32:05.224646 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:05.224563    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e08c1ee-66f2-4f91-974c-5dc7d710d35d" path="/var/lib/kubelet/pods/9e08c1ee-66f2-4f91-974c-5dc7d710d35d/volumes"
Nov 02 02:32:10.844689 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:10.844645    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:11.021045 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:11.021010    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:11.407200 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:11.407165    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:11.579507 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:11.579466    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:11.777778 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:11.777682    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:12.137928 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:32:12.137894    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:32:15.105123 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:15.105043    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2" podUID="52647b8b-755e-4e68-b331-ad5c0419034d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:32:15.105592 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:15.105160    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86" podUID="c4532c7e-35a4-4b12-b258-cafabc53e5e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:32:16.731701 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:16.731661    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:21.023995 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.023928    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:32:21.117921 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.117895    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:32:21.134206 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.134172    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52647b8b-755e-4e68-b331-ad5c0419034d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52647b8b-755e-4e68-b331-ad5c0419034d-plugins-config-volume\") pod \"52647b8b-755e-4e68-b331-ad5c0419034d\" (UID: \"52647b8b-755e-4e68-b331-ad5c0419034d\") "
Nov 02 02:32:21.134360 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.134261    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52647b8b-755e-4e68-b331-ad5c0419034d-kube-api-access-lbc92\" (UniqueName: \"kubernetes.io/projected/52647b8b-755e-4e68-b331-ad5c0419034d-kube-api-access-lbc92\") pod \"52647b8b-755e-4e68-b331-ad5c0419034d\" (UID: \"52647b8b-755e-4e68-b331-ad5c0419034d\") "
Nov 02 02:32:21.134622 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.134598    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52647b8b-755e-4e68-b331-ad5c0419034d-plugins-config-volume" pod "52647b8b-755e-4e68-b331-ad5c0419034d" (UID: "52647b8b-755e-4e68-b331-ad5c0419034d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:32:21.136831 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.136811    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52647b8b-755e-4e68-b331-ad5c0419034d-kube-api-access-lbc92" pod "52647b8b-755e-4e68-b331-ad5c0419034d" (UID: "52647b8b-755e-4e68-b331-ad5c0419034d"). InnerVolumeSpecName "kube-api-access-lbc92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:21.235189 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.235163    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4532c7e-35a4-4b12-b258-cafabc53e5e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4532c7e-35a4-4b12-b258-cafabc53e5e7-plugins-config-volume\") pod \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\" (UID: \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\") "
Nov 02 02:32:21.235305 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.235234    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4532c7e-35a4-4b12-b258-cafabc53e5e7-kube-api-access-x99hb\" (UniqueName: \"kubernetes.io/projected/c4532c7e-35a4-4b12-b258-cafabc53e5e7-kube-api-access-x99hb\") pod \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\" (UID: \"c4532c7e-35a4-4b12-b258-cafabc53e5e7\") "
Nov 02 02:32:21.235450 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.235438    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52647b8b-755e-4e68-b331-ad5c0419034d-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:21.235491 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.235452    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbc92\" (UniqueName: \"kubernetes.io/projected/52647b8b-755e-4e68-b331-ad5c0419034d-kube-api-access-lbc92\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:21.235543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.235485    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4532c7e-35a4-4b12-b258-cafabc53e5e7-plugins-config-volume" pod "c4532c7e-35a4-4b12-b258-cafabc53e5e7" (UID: "c4532c7e-35a4-4b12-b258-cafabc53e5e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:32:21.237391 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.237372    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4532c7e-35a4-4b12-b258-cafabc53e5e7-kube-api-access-x99hb" pod "c4532c7e-35a4-4b12-b258-cafabc53e5e7" (UID: "c4532c7e-35a4-4b12-b258-cafabc53e5e7"). InnerVolumeSpecName "kube-api-access-x99hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:21.336962 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.336873    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4532c7e-35a4-4b12-b258-cafabc53e5e7-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:21.336962 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.336910    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x99hb\" (UniqueName: \"kubernetes.io/projected/c4532c7e-35a4-4b12-b258-cafabc53e5e7-kube-api-access-x99hb\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:21.882011 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.881967    4912 generic.go:356] "Generic (PLEG): container finished" podID="52647b8b-755e-4e68-b331-ad5c0419034d" containerID="719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0" exitCode=0
Nov 02 02:32:21.882255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.882019    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"
Nov 02 02:32:21.882255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.882073    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2" event={"ID":"52647b8b-755e-4e68-b331-ad5c0419034d","Type":"ContainerDied","Data":"719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0"}
Nov 02 02:32:21.882255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.882131    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2" event={"ID":"52647b8b-755e-4e68-b331-ad5c0419034d","Type":"ContainerDied","Data":"2886fc00732790f7be4fce4ac62e3d0214607645766bf6e5b407f72d858d1981"}
Nov 02 02:32:21.882255 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.882159    4912 scope.go:122] "RemoveContainer" containerID="719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0"
Nov 02 02:32:21.883713 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.883694    4912 generic.go:356] "Generic (PLEG): container finished" podID="c4532c7e-35a4-4b12-b258-cafabc53e5e7" containerID="599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05" exitCode=0
Nov 02 02:32:21.883838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.883737    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86" event={"ID":"c4532c7e-35a4-4b12-b258-cafabc53e5e7","Type":"ContainerDied","Data":"599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05"}
Nov 02 02:32:21.883838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.883742    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"
Nov 02 02:32:21.883838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.883770    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86" event={"ID":"c4532c7e-35a4-4b12-b258-cafabc53e5e7","Type":"ContainerDied","Data":"7eebd73cf700d50ac97da82cf080f334d6c998c3d14e6451d3e9f0ea2a88fa83"}
Nov 02 02:32:21.896499 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.896477    4912 scope.go:122] "RemoveContainer" containerID="719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0"
Nov 02 02:32:21.896910 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:32:21.896890    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0\": container with ID starting with 719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0 not found: ID does not exist" containerID="719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0"
Nov 02 02:32:21.896968 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.896921    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0"} err="failed to get container status \"719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0\": rpc error: code = NotFound desc = could not find container \"719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0\": container with ID starting with 719d70272ff0a273bc38956ef2fa0b5f2ee4106f4626378bee69087aeea93af0 not found: ID does not exist"
Nov 02 02:32:21.896968 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.896938    4912 scope.go:122] "RemoveContainer" containerID="599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05"
Nov 02 02:32:21.906174 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.906155    4912 scope.go:122] "RemoveContainer" containerID="599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05"
Nov 02 02:32:21.906452 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:32:21.906434    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05\": container with ID starting with 599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05 not found: ID does not exist" containerID="599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05"
Nov 02 02:32:21.906517 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:21.906462    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05"} err="failed to get container status \"599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05\": rpc error: code = NotFound desc = could not find container \"599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05\": container with ID starting with 599cb2a16f6290c3c4b5d66ba1cc7bb793f129d38f3df47402b300c931b37f05 not found: ID does not exist"
Nov 02 02:32:22.049478 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:22.049382    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"]
Nov 02 02:32:22.098866 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:22.098824    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-18df32-769f8lgh86"]
Nov 02 02:32:22.161247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:22.161206    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"]
Nov 02 02:32:22.165603 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:22.165576    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-18df32-557f69b986gzx2"]
Nov 02 02:32:23.223789 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:23.223756    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52647b8b-755e-4e68-b331-ad5c0419034d" path="/var/lib/kubelet/pods/52647b8b-755e-4e68-b331-ad5c0419034d/volumes"
Nov 02 02:32:23.224196 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:23.224168    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4532c7e-35a4-4b12-b258-cafabc53e5e7" path="/var/lib/kubelet/pods/c4532c7e-35a4-4b12-b258-cafabc53e5e7/volumes"
Nov 02 02:32:30.555773 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:30.555692    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:42.137526 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:32:42.137492    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:32:43.047783 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:43.047747    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"]
Nov 02 02:32:43.098543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:43.098505    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsdbt"]
Nov 02 02:32:43.225146 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:43.225115    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f607c38-aea4-4d18-8605-1159c379df08" path="/var/lib/kubelet/pods/2f607c38-aea4-4d18-8605-1159c379df08/volumes"
Nov 02 02:32:50.046913 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:50.046882    4912 scope.go:122] "RemoveContainer" containerID="b8b5894b972e46d34894986db89b180f06ed04966d92a7172a5a434f65290be6"
Nov 02 02:32:56.607680 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:56.607640    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:32:59.076120 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:32:59.076046    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:33:04.418708 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:04.418209    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4tqpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:33:04.418708 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:04.418283    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podUID="ea71fe46-235c-43bb-adc6-708430a55abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:33:12.137097 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:33:12.137038    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:33:25.615912 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.615876    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"]
Nov 02 02:33:25.616387 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.616370    4912 cpu_manager.go:414] "containerMap: removing container" podUID="c4532c7e-35a4-4b12-b258-cafabc53e5e7" containerName="payload-pre-processing"
Nov 02 02:33:25.616439 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.616387    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4532c7e-35a4-4b12-b258-cafabc53e5e7" containerName="payload-pre-processing"
Nov 02 02:33:25.616439 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.616399    4912 cpu_manager.go:414] "containerMap: removing container" podUID="52647b8b-755e-4e68-b331-ad5c0419034d" containerName="payload-processing"
Nov 02 02:33:25.616439 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.616404    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52647b8b-755e-4e68-b331-ad5c0419034d" containerName="payload-processing"
Nov 02 02:33:25.616439 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.616432    4912 cpu_manager.go:414] "containerMap: removing container" podUID="9e08c1ee-66f2-4f91-974c-5dc7d710d35d" containerName="maas-api"
Nov 02 02:33:25.616439 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.616437    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e08c1ee-66f2-4f91-974c-5dc7d710d35d" containerName="maas-api"
Nov 02 02:33:25.616616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.616509    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4532c7e-35a4-4b12-b258-cafabc53e5e7" containerName="payload-pre-processing"
Nov 02 02:33:25.616616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.616517    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="52647b8b-755e-4e68-b331-ad5c0419034d" containerName="payload-processing"
Nov 02 02:33:25.616616 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.616527    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e08c1ee-66f2-4f91-974c-5dc7d710d35d" containerName="maas-api"
Nov 02 02:33:25.619820 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.619800    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.673350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.673313    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:33:25.676855 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.676825    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"]
Nov 02 02:33:25.723246 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.723218    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:33:25.734408 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.734382    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvrm\" (UniqueName: \"kubernetes.io/projected/a55a8923-fa38-4667-aa16-0e42f6b12644-kube-api-access-fzvrm\") pod \"maas-api-revoke-sub-452b6d7b-p9r7x\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.734519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.734503    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9r7x\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.734569 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.734532    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9r7x\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.835526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.835490    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9r7x\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.835526 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.835534    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9r7x\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.835754 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.835623    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvrm\" (UniqueName: \"kubernetes.io/projected/a55a8923-fa38-4667-aa16-0e42f6b12644-kube-api-access-fzvrm\") pod \"maas-api-revoke-sub-452b6d7b-p9r7x\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.836296 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.836271    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9r7x\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.838109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.838089    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9r7x\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.896484 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.896411    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvrm\" (UniqueName: \"kubernetes.io/projected/a55a8923-fa38-4667-aa16-0e42f6b12644-kube-api-access-fzvrm\") pod \"maas-api-revoke-sub-452b6d7b-p9r7x\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:25.934107 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:25.934075    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:26.318954 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:26.318926    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"]
Nov 02 02:33:26.320950 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:33:26.320919    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55a8923_fa38_4667_aa16_0e42f6b12644.slice/crio-0c32fb501c11b9f5eeff891f6741690471b41b1ef82cce2e778e66d494cec6bc WatchSource:0}: Error finding container 0c32fb501c11b9f5eeff891f6741690471b41b1ef82cce2e778e66d494cec6bc: Status 404 returned error can't find the container with id 0c32fb501c11b9f5eeff891f6741690471b41b1ef82cce2e778e66d494cec6bc
Nov 02 02:33:26.361710 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:26.361682    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x" event={"ID":"a55a8923-fa38-4667-aa16-0e42f6b12644","Type":"ContainerStarted","Data":"0c32fb501c11b9f5eeff891f6741690471b41b1ef82cce2e778e66d494cec6bc"}
Nov 02 02:33:27.369568 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:27.369536    4912 generic.go:356] "Generic (PLEG): container finished" podID="a55a8923-fa38-4667-aa16-0e42f6b12644" containerID="b44569f06f7e21e20fc5e00719095fcafa15481bd6249533f6131b112768696c" exitCode=0
Nov 02 02:33:27.369968 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:27.369623    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x" event={"ID":"a55a8923-fa38-4667-aa16-0e42f6b12644","Type":"ContainerDied","Data":"b44569f06f7e21e20fc5e00719095fcafa15481bd6249533f6131b112768696c"}
Nov 02 02:33:28.698954 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.698638    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:28.775482 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.775442    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-openshift-ca\") pod \"a55a8923-fa38-4667-aa16-0e42f6b12644\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") "
Nov 02 02:33:28.775788 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.775748    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-xks-ca\") pod \"a55a8923-fa38-4667-aa16-0e42f6b12644\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") "
Nov 02 02:33:28.776415 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.776378    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55a8923-fa38-4667-aa16-0e42f6b12644-kube-api-access-fzvrm\" (UniqueName: \"kubernetes.io/projected/a55a8923-fa38-4667-aa16-0e42f6b12644-kube-api-access-fzvrm\") pod \"a55a8923-fa38-4667-aa16-0e42f6b12644\" (UID: \"a55a8923-fa38-4667-aa16-0e42f6b12644\") "
Nov 02 02:33:28.778014 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.777990    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-openshift-ca" pod "a55a8923-fa38-4667-aa16-0e42f6b12644" (UID: "a55a8923-fa38-4667-aa16-0e42f6b12644"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:33:28.788359 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.788311    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55a8923-fa38-4667-aa16-0e42f6b12644-kube-api-access-fzvrm" pod "a55a8923-fa38-4667-aa16-0e42f6b12644" (UID: "a55a8923-fa38-4667-aa16-0e42f6b12644"). InnerVolumeSpecName "kube-api-access-fzvrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:33:28.808893 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.808845    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-xks-ca" pod "a55a8923-fa38-4667-aa16-0e42f6b12644" (UID: "a55a8923-fa38-4667-aa16-0e42f6b12644"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:33:28.882155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.882045    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:33:28.882155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.882103    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55a8923-fa38-4667-aa16-0e42f6b12644-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:33:28.882155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:28.882117    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvrm\" (UniqueName: \"kubernetes.io/projected/a55a8923-fa38-4667-aa16-0e42f6b12644-kube-api-access-fzvrm\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:33:29.409884 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:29.409842    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x" event={"ID":"a55a8923-fa38-4667-aa16-0e42f6b12644","Type":"ContainerDied","Data":"0c32fb501c11b9f5eeff891f6741690471b41b1ef82cce2e778e66d494cec6bc"}
Nov 02 02:33:29.410305 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:29.410284    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c32fb501c11b9f5eeff891f6741690471b41b1ef82cce2e778e66d494cec6bc"
Nov 02 02:33:29.410468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:29.410216    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"
Nov 02 02:33:38.453311 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:38.453273    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:33:41.837591 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:41.837548    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:33:42.066777 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:42.066739    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:33:42.137633 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:33:42.137598    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:33:45.137632 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:45.137594    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:33:49.273164 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.273129    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"]
Nov 02 02:33:49.273636 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.273620    4912 cpu_manager.go:414] "containerMap: removing container" podUID="a55a8923-fa38-4667-aa16-0e42f6b12644" containerName="revoke-keys"
Nov 02 02:33:49.273681 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.273638    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55a8923-fa38-4667-aa16-0e42f6b12644" containerName="revoke-keys"
Nov 02 02:33:49.273726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.273715    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55a8923-fa38-4667-aa16-0e42f6b12644" containerName="revoke-keys"
Nov 02 02:33:49.277162 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.277142    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.325291 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.325261    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"]
Nov 02 02:33:49.369969 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.369930    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:33:49.370179 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.370044    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:33:49.370571 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.370551    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-srwcn\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.370618 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.370581    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wr4\" (UniqueName: \"kubernetes.io/projected/6e256dff-e6e1-4589-8d02-12692ff79b61-kube-api-access-j4wr4\") pod \"maas-api-revoke-sub-3e31a506-srwcn\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.370707 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.370695    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-srwcn\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.471821 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.471779    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-srwcn\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.472039 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.471853    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-srwcn\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.472039 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.471873    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wr4\" (UniqueName: \"kubernetes.io/projected/6e256dff-e6e1-4589-8d02-12692ff79b61-kube-api-access-j4wr4\") pod \"maas-api-revoke-sub-3e31a506-srwcn\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.472461 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.472435    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-srwcn\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.474580 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.474549    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-srwcn\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.533210 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.533140    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wr4\" (UniqueName: \"kubernetes.io/projected/6e256dff-e6e1-4589-8d02-12692ff79b61-kube-api-access-j4wr4\") pod \"maas-api-revoke-sub-3e31a506-srwcn\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.591714 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.591675    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:49.971174 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:49.969991    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"]
Nov 02 02:33:49.972209 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:33:49.972174    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e256dff_e6e1_4589_8d02_12692ff79b61.slice/crio-e0f15660164d75b18a16522518a850e07c44fb5eb0d10284598354e980b849b4 WatchSource:0}: Error finding container e0f15660164d75b18a16522518a850e07c44fb5eb0d10284598354e980b849b4: Status 404 returned error can't find the container with id e0f15660164d75b18a16522518a850e07c44fb5eb0d10284598354e980b849b4
Nov 02 02:33:50.086434 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:50.086400    4912 scope.go:122] "RemoveContainer" containerID="9cdb24f43d6a6d29cafbcad9923c343256d94eaf20ae700efde4fbab33b600b5"
Nov 02 02:33:50.571839 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:50.571112    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn" event={"ID":"6e256dff-e6e1-4589-8d02-12692ff79b61","Type":"ContainerStarted","Data":"f4e81b9ee6f71e040d49b4cd2f7993c1ff4aa2fe745766e1c8956a9fd7c0c60c"}
Nov 02 02:33:50.571839 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:50.571186    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn" event={"ID":"6e256dff-e6e1-4589-8d02-12692ff79b61","Type":"ContainerStarted","Data":"e0f15660164d75b18a16522518a850e07c44fb5eb0d10284598354e980b849b4"}
Nov 02 02:33:50.688783 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:50.688717    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn" podStartSLOduration=1.688699863 podStartE2EDuration="1.688699863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:33:50.686445335 +0000 UTC m=+3723.591910693" watchObservedRunningTime="2026-11-02 02:33:50.688699863 +0000 UTC m=+3723.594165217"
Nov 02 02:33:51.622117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:51.621342    4912 generic.go:356] "Generic (PLEG): container finished" podID="6e256dff-e6e1-4589-8d02-12692ff79b61" containerID="f4e81b9ee6f71e040d49b4cd2f7993c1ff4aa2fe745766e1c8956a9fd7c0c60c" exitCode=0
Nov 02 02:33:51.622117 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:51.621548    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn" event={"ID":"6e256dff-e6e1-4589-8d02-12692ff79b61","Type":"ContainerDied","Data":"f4e81b9ee6f71e040d49b4cd2f7993c1ff4aa2fe745766e1c8956a9fd7c0c60c"}
Nov 02 02:33:52.761765 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:52.761744    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:52.938392 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:52.938313    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-xks-ca\") pod \"6e256dff-e6e1-4589-8d02-12692ff79b61\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") "
Nov 02 02:33:52.938545 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:52.938409    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e256dff-e6e1-4589-8d02-12692ff79b61-kube-api-access-j4wr4\" (UniqueName: \"kubernetes.io/projected/6e256dff-e6e1-4589-8d02-12692ff79b61-kube-api-access-j4wr4\") pod \"6e256dff-e6e1-4589-8d02-12692ff79b61\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") "
Nov 02 02:33:52.938545 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:52.938460    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-openshift-ca\") pod \"6e256dff-e6e1-4589-8d02-12692ff79b61\" (UID: \"6e256dff-e6e1-4589-8d02-12692ff79b61\") "
Nov 02 02:33:52.938849 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:52.938823    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-openshift-ca" pod "6e256dff-e6e1-4589-8d02-12692ff79b61" (UID: "6e256dff-e6e1-4589-8d02-12692ff79b61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:33:52.940505 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:52.940478    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-xks-ca" pod "6e256dff-e6e1-4589-8d02-12692ff79b61" (UID: "6e256dff-e6e1-4589-8d02-12692ff79b61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:33:52.940625 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:52.940613    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e256dff-e6e1-4589-8d02-12692ff79b61-kube-api-access-j4wr4" pod "6e256dff-e6e1-4589-8d02-12692ff79b61" (UID: "6e256dff-e6e1-4589-8d02-12692ff79b61"). InnerVolumeSpecName "kube-api-access-j4wr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:33:53.047657 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:53.040893    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4wr4\" (UniqueName: \"kubernetes.io/projected/6e256dff-e6e1-4589-8d02-12692ff79b61-kube-api-access-j4wr4\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:33:53.047657 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:53.040937    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:33:53.047657 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:53.040953    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e256dff-e6e1-4589-8d02-12692ff79b61-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:33:53.635004 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:53.634966    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn" event={"ID":"6e256dff-e6e1-4589-8d02-12692ff79b61","Type":"ContainerDied","Data":"e0f15660164d75b18a16522518a850e07c44fb5eb0d10284598354e980b849b4"}
Nov 02 02:33:53.635004 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:53.634987    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"
Nov 02 02:33:53.635249 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:53.635015    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0f15660164d75b18a16522518a850e07c44fb5eb0d10284598354e980b849b4"
Nov 02 02:33:59.373453 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:33:59.373372    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:04.422276 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:04.422223    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-4tqpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:34:04.422932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:04.422901    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-4tqpd" podUID="ea71fe46-235c-43bb-adc6-708430a55abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:34:12.138753 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:34:12.138714    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:34:15.714444 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.714412    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"]
Nov 02 02:34:15.714901 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.714888    4912 cpu_manager.go:414] "containerMap: removing container" podUID="6e256dff-e6e1-4589-8d02-12692ff79b61" containerName="revoke-keys"
Nov 02 02:34:15.714953 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.714901    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e256dff-e6e1-4589-8d02-12692ff79b61" containerName="revoke-keys"
Nov 02 02:34:15.715000 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.714990    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e256dff-e6e1-4589-8d02-12692ff79b61" containerName="revoke-keys"
Nov 02 02:34:15.718298 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.718275    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:15.774021 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.773981    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"]
Nov 02 02:34:15.817853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.817820    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b181ed9a-2t85f\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:15.818047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.817976    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs92l\" (UniqueName: \"kubernetes.io/projected/ad197f27-bbbd-4f75-a89c-71ccc59343b2-kube-api-access-fs92l\") pod \"maas-api-revoke-sub-b181ed9a-2t85f\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:15.818047 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.818039    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b181ed9a-2t85f\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:15.819040 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.819020    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:34:15.819144 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.819136    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:34:15.919156 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.919118    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs92l\" (UniqueName: \"kubernetes.io/projected/ad197f27-bbbd-4f75-a89c-71ccc59343b2-kube-api-access-fs92l\") pod \"maas-api-revoke-sub-b181ed9a-2t85f\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:15.919378 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.919172    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b181ed9a-2t85f\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:15.919378 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.919233    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b181ed9a-2t85f\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:15.919841 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.919820    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b181ed9a-2t85f\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:15.921900 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.921877    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b181ed9a-2t85f\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:15.979947 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:15.979877    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs92l\" (UniqueName: \"kubernetes.io/projected/ad197f27-bbbd-4f75-a89c-71ccc59343b2-kube-api-access-fs92l\") pod \"maas-api-revoke-sub-b181ed9a-2t85f\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:16.031788 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:16.031753    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:16.404885 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:16.404858    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"]
Nov 02 02:34:16.406461 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:34:16.406430    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad197f27_bbbd_4f75_a89c_71ccc59343b2.slice/crio-4a823f19647c18b9b0c26c8290c5e974be25ad3cd31ce8b474a5b60cab3869c7 WatchSource:0}: Error finding container 4a823f19647c18b9b0c26c8290c5e974be25ad3cd31ce8b474a5b60cab3869c7: Status 404 returned error can't find the container with id 4a823f19647c18b9b0c26c8290c5e974be25ad3cd31ce8b474a5b60cab3869c7
Nov 02 02:34:16.815418 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:16.815373    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f" event={"ID":"ad197f27-bbbd-4f75-a89c-71ccc59343b2","Type":"ContainerStarted","Data":"babbb793691b4372b1a5a0554032f9f0389e082cb413c312de8f4f4d1c0b36bd"}
Nov 02 02:34:16.815836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:16.815431    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f" event={"ID":"ad197f27-bbbd-4f75-a89c-71ccc59343b2","Type":"ContainerStarted","Data":"4a823f19647c18b9b0c26c8290c5e974be25ad3cd31ce8b474a5b60cab3869c7"}
Nov 02 02:34:16.931674 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:16.931580    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f" podStartSLOduration=1.93156645 podStartE2EDuration="1.93156645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:34:16.929842233 +0000 UTC m=+3749.835307593" watchObservedRunningTime="2026-11-02 02:34:16.93156645 +0000 UTC m=+3749.837031807"
Nov 02 02:34:17.822667 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:17.822631    4912 generic.go:356] "Generic (PLEG): container finished" podID="ad197f27-bbbd-4f75-a89c-71ccc59343b2" containerID="babbb793691b4372b1a5a0554032f9f0389e082cb413c312de8f4f4d1c0b36bd" exitCode=0
Nov 02 02:34:17.823140 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:17.822714    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f" event={"ID":"ad197f27-bbbd-4f75-a89c-71ccc59343b2","Type":"ContainerDied","Data":"babbb793691b4372b1a5a0554032f9f0389e082cb413c312de8f4f4d1c0b36bd"}
Nov 02 02:34:18.972636 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:18.972598    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:19.051162 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.051129    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-xks-ca\") pod \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") "
Nov 02 02:34:19.051343 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.051244    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad197f27-bbbd-4f75-a89c-71ccc59343b2-kube-api-access-fs92l\" (UniqueName: \"kubernetes.io/projected/ad197f27-bbbd-4f75-a89c-71ccc59343b2-kube-api-access-fs92l\") pod \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") "
Nov 02 02:34:19.051407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.051347    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-openshift-ca\") pod \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\" (UID: \"ad197f27-bbbd-4f75-a89c-71ccc59343b2\") "
Nov 02 02:34:19.051726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.051703    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-openshift-ca" pod "ad197f27-bbbd-4f75-a89c-71ccc59343b2" (UID: "ad197f27-bbbd-4f75-a89c-71ccc59343b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:34:19.053467 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.053445    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-xks-ca" pod "ad197f27-bbbd-4f75-a89c-71ccc59343b2" (UID: "ad197f27-bbbd-4f75-a89c-71ccc59343b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:34:19.053557 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.053515    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad197f27-bbbd-4f75-a89c-71ccc59343b2-kube-api-access-fs92l" pod "ad197f27-bbbd-4f75-a89c-71ccc59343b2" (UID: "ad197f27-bbbd-4f75-a89c-71ccc59343b2"). InnerVolumeSpecName "kube-api-access-fs92l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:34:19.152289 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.152261    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-openshift-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:19.152289 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.152290    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad197f27-bbbd-4f75-a89c-71ccc59343b2-maas-api-xks-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:19.152552 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.152303    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs92l\" (UniqueName: \"kubernetes.io/projected/ad197f27-bbbd-4f75-a89c-71ccc59343b2-kube-api-access-fs92l\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:19.836905 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.836873    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"
Nov 02 02:34:19.837109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.836903    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f" event={"ID":"ad197f27-bbbd-4f75-a89c-71ccc59343b2","Type":"ContainerDied","Data":"4a823f19647c18b9b0c26c8290c5e974be25ad3cd31ce8b474a5b60cab3869c7"}
Nov 02 02:34:19.837109 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:19.836977    4912 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a823f19647c18b9b0c26c8290c5e974be25ad3cd31ce8b474a5b60cab3869c7"
Nov 02 02:34:28.333203 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:28.333159    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:29.016504 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.016464    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"]
Nov 02 02:34:29.017207 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.017190    4912 cpu_manager.go:414] "containerMap: removing container" podUID="ad197f27-bbbd-4f75-a89c-71ccc59343b2" containerName="revoke-keys"
Nov 02 02:34:29.017284 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.017209    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad197f27-bbbd-4f75-a89c-71ccc59343b2" containerName="revoke-keys"
Nov 02 02:34:29.017360 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.017348    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad197f27-bbbd-4f75-a89c-71ccc59343b2" containerName="revoke-keys"
Nov 02 02:34:29.021258 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.021237    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.057969 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.057926    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slq6f\" (UniqueName: \"kubernetes.io/projected/0d45d7c5-9edd-465f-a034-7c483d43d47f-kube-api-access-slq6f\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.058177 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.058010    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-metrics-certs\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.058177 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.058087    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-maas-api-tls\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.058267 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.058205    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d45d7c5-9edd-465f-a034-7c483d43d47f-openshift-service-ca\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.123421 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.123332    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 02:34:29.123619 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.123423    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 02:34:29.123619 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.123442    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2n4c5\""
Nov 02 02:34:29.127041 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.127016    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"]
Nov 02 02:34:29.158738 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.158713    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d45d7c5-9edd-465f-a034-7c483d43d47f-openshift-service-ca\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.158903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.158768    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slq6f\" (UniqueName: \"kubernetes.io/projected/0d45d7c5-9edd-465f-a034-7c483d43d47f-kube-api-access-slq6f\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.158903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.158816    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-metrics-certs\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.158903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.158864    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-maas-api-tls\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.159527 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.159495    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d45d7c5-9edd-465f-a034-7c483d43d47f-openshift-service-ca\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.161991 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.161966    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-maas-api-tls\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.162146 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.161976    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-metrics-certs\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.219862 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.219825    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slq6f\" (UniqueName: \"kubernetes.io/projected/0d45d7c5-9edd-465f-a034-7c483d43d47f-kube-api-access-slq6f\") pod \"maas-api-66b669b4fd-7fv92\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") " pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.336401 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.336320    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:29.716558 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.716527    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"]
Nov 02 02:34:29.717474 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:34:29.717443    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d45d7c5_9edd_465f_a034_7c483d43d47f.slice/crio-c2f70ea8b6030ab96e3b2d2ad1c8f118005ba674b1b26825b033738c5fde939e WatchSource:0}: Error finding container c2f70ea8b6030ab96e3b2d2ad1c8f118005ba674b1b26825b033738c5fde939e: Status 404 returned error can't find the container with id c2f70ea8b6030ab96e3b2d2ad1c8f118005ba674b1b26825b033738c5fde939e
Nov 02 02:34:29.719036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.719016    4912 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:34:29.902072 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:29.902020    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92" event={"ID":"0d45d7c5-9edd-465f-a034-7c483d43d47f","Type":"ContainerStarted","Data":"c2f70ea8b6030ab96e3b2d2ad1c8f118005ba674b1b26825b033738c5fde939e"}
Nov 02 02:34:30.910717 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:30.910624    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92" event={"ID":"0d45d7c5-9edd-465f-a034-7c483d43d47f","Type":"ContainerStarted","Data":"c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6"}
Nov 02 02:34:30.911119 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:30.910748    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:31.027118 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:31.027048    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92" podStartSLOduration=2.104503034 podStartE2EDuration="3.027033951s" totalImagesPullingTime="922.530917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:34:31.026534999 +0000 UTC m=+3763.932000347" watchObservedRunningTime="2026-11-02 02:34:31.027033951 +0000 UTC m=+3763.932499309"
Nov 02 02:34:35.590721 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:35.590673    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:36.050348 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:36.050309    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"]
Nov 02 02:34:36.099317 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:36.099278    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bpgtg"]
Nov 02 02:34:36.196581 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:36.196544    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:36.426166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:36.426129    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:36.922797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:36.922763    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:37.225490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.225418    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954a54e4-91b9-499f-9c6b-e9311664f73f" path="/var/lib/kubelet/pods/954a54e4-91b9-499f-9c6b-e9311664f73f/volumes"
Nov 02 02:34:37.492626 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.492541    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"]
Nov 02 02:34:37.492793 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.492739    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92" podUID="0d45d7c5-9edd-465f-a034-7c483d43d47f" containerName="maas-api" containerID="cri-o://c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6" gracePeriod=30
Nov 02 02:34:37.526669 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.526639    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"]
Nov 02 02:34:37.530411 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.530389    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.590211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.590183    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"]
Nov 02 02:34:37.644800 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.644771    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsmw\" (UniqueName: \"kubernetes.io/projected/439c3d1f-470d-4ec6-a72d-13c031097a64-kube-api-access-mqsmw\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.644951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.644806    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/439c3d1f-470d-4ec6-a72d-13c031097a64-openshift-service-ca\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.644951 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.644836    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/439c3d1f-470d-4ec6-a72d-13c031097a64-metrics-certs\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.645045 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.644984    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/439c3d1f-470d-4ec6-a72d-13c031097a64-maas-api-tls\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.745636 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.745575    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsmw\" (UniqueName: \"kubernetes.io/projected/439c3d1f-470d-4ec6-a72d-13c031097a64-kube-api-access-mqsmw\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.745636 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.745612    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/439c3d1f-470d-4ec6-a72d-13c031097a64-openshift-service-ca\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.745830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.745649    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/439c3d1f-470d-4ec6-a72d-13c031097a64-metrics-certs\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.745830 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.745695    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/439c3d1f-470d-4ec6-a72d-13c031097a64-maas-api-tls\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.746406 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.746380    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/439c3d1f-470d-4ec6-a72d-13c031097a64-openshift-service-ca\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.748229 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.748208    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/439c3d1f-470d-4ec6-a72d-13c031097a64-metrics-certs\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.748339 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.748310    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/439c3d1f-470d-4ec6-a72d-13c031097a64-maas-api-tls\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.769587 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.769564    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:37.806602 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.806569    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsmw\" (UniqueName: \"kubernetes.io/projected/439c3d1f-470d-4ec6-a72d-13c031097a64-kube-api-access-mqsmw\") pod \"maas-api-6d4c89cd86-lfrzw\" (UID: \"439c3d1f-470d-4ec6-a72d-13c031097a64\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.847103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.847075    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-metrics-certs\") pod \"0d45d7c5-9edd-465f-a034-7c483d43d47f\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") "
Nov 02 02:34:37.847216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.847111    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d45d7c5-9edd-465f-a034-7c483d43d47f-kube-api-access-slq6f\" (UniqueName: \"kubernetes.io/projected/0d45d7c5-9edd-465f-a034-7c483d43d47f-kube-api-access-slq6f\") pod \"0d45d7c5-9edd-465f-a034-7c483d43d47f\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") "
Nov 02 02:34:37.847216 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.847191    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-maas-api-tls\") pod \"0d45d7c5-9edd-465f-a034-7c483d43d47f\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") "
Nov 02 02:34:37.847304 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.847284    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d45d7c5-9edd-465f-a034-7c483d43d47f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d45d7c5-9edd-465f-a034-7c483d43d47f-openshift-service-ca\") pod \"0d45d7c5-9edd-465f-a034-7c483d43d47f\" (UID: \"0d45d7c5-9edd-465f-a034-7c483d43d47f\") "
Nov 02 02:34:37.847731 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.847702    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d45d7c5-9edd-465f-a034-7c483d43d47f-openshift-service-ca" pod "0d45d7c5-9edd-465f-a034-7c483d43d47f" (UID: "0d45d7c5-9edd-465f-a034-7c483d43d47f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:34:37.849365 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.849337    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-metrics-certs" pod "0d45d7c5-9edd-465f-a034-7c483d43d47f" (UID: "0d45d7c5-9edd-465f-a034-7c483d43d47f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:34:37.849501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.849439    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d45d7c5-9edd-465f-a034-7c483d43d47f-kube-api-access-slq6f" pod "0d45d7c5-9edd-465f-a034-7c483d43d47f" (UID: "0d45d7c5-9edd-465f-a034-7c483d43d47f"). InnerVolumeSpecName "kube-api-access-slq6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:34:37.849501 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.849450    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-maas-api-tls" pod "0d45d7c5-9edd-465f-a034-7c483d43d47f" (UID: "0d45d7c5-9edd-465f-a034-7c483d43d47f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:34:37.878678 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.878654    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:37.961989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.953460    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d45d7c5-9edd-465f-a034-7c483d43d47f-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:37.961989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.953496    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:37.961989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.953512    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slq6f\" (UniqueName: \"kubernetes.io/projected/0d45d7c5-9edd-465f-a034-7c483d43d47f-kube-api-access-slq6f\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:37.961989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.953528    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d45d7c5-9edd-465f-a034-7c483d43d47f-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:34:37.961989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.959496    4912 generic.go:356] "Generic (PLEG): container finished" podID="0d45d7c5-9edd-465f-a034-7c483d43d47f" containerID="c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6" exitCode=0
Nov 02 02:34:37.961989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.959539    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92" event={"ID":"0d45d7c5-9edd-465f-a034-7c483d43d47f","Type":"ContainerDied","Data":"c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6"}
Nov 02 02:34:37.961989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.959599    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92" event={"ID":"0d45d7c5-9edd-465f-a034-7c483d43d47f","Type":"ContainerDied","Data":"c2f70ea8b6030ab96e3b2d2ad1c8f118005ba674b1b26825b033738c5fde939e"}
Nov 02 02:34:37.961989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.959620    4912 scope.go:122] "RemoveContainer" containerID="c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6"
Nov 02 02:34:37.961989 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.959735    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"
Nov 02 02:34:37.975974 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.975118    4912 scope.go:122] "RemoveContainer" containerID="c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6"
Nov 02 02:34:37.976681 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:34:37.976649    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6\": container with ID starting with c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6 not found: ID does not exist" containerID="c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6"
Nov 02 02:34:37.976790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:37.976694    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6"} err="failed to get container status \"c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6\": rpc error: code = NotFound desc = could not find container \"c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6\": container with ID starting with c000ba0c2a86f642220659b4ccf2f9373f3c682c4c5aed72f5a3f13803ab81d6 not found: ID does not exist"
Nov 02 02:34:38.127464 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:38.127433    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"]
Nov 02 02:34:38.130831 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:38.130804    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b669b4fd-7fv92"]
Nov 02 02:34:38.259777 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:38.259750    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"]
Nov 02 02:34:38.261034 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:34:38.261006    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439c3d1f_470d_4ec6_a72d_13c031097a64.slice/crio-b7367309e71726356538b11c6c0add82cf90db3ab63cbd3b8c2142e64d0410ff WatchSource:0}: Error finding container b7367309e71726356538b11c6c0add82cf90db3ab63cbd3b8c2142e64d0410ff: Status 404 returned error can't find the container with id b7367309e71726356538b11c6c0add82cf90db3ab63cbd3b8c2142e64d0410ff
Nov 02 02:34:38.967246 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:38.967207    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw" event={"ID":"439c3d1f-470d-4ec6-a72d-13c031097a64","Type":"ContainerStarted","Data":"b7367309e71726356538b11c6c0add82cf90db3ab63cbd3b8c2142e64d0410ff"}
Nov 02 02:34:39.224559 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:39.224481    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d45d7c5-9edd-465f-a034-7c483d43d47f" path="/var/lib/kubelet/pods/0d45d7c5-9edd-465f-a034-7c483d43d47f/volumes"
Nov 02 02:34:39.750551 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:39.750515    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:39.940104 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:39.940041    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:39.977665 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:39.977614    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw" event={"ID":"439c3d1f-470d-4ec6-a72d-13c031097a64","Type":"ContainerStarted","Data":"21f7990c706792c3974e4f59848a3e15aa5d4760b6026feefaef1ea3fa4f5859"}
Nov 02 02:34:39.978189 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:39.977707    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:40.097155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:40.097037    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw" podStartSLOduration=2.297434619 podStartE2EDuration="3.097022665s" totalImagesPullingTime="799.588046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:34:40.094505496 +0000 UTC m=+3772.999970858" watchObservedRunningTime="2026-11-02 02:34:40.097022665 +0000 UTC m=+3773.002488015"
Nov 02 02:34:42.138668 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:34:42.138625    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:34:42.977126 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:42.977086    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:43.215186 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.215149    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:43.361315 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.361230    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"]
Nov 02 02:34:43.361838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.361818    4912 cpu_manager.go:414] "containerMap: removing container" podUID="0d45d7c5-9edd-465f-a034-7c483d43d47f" containerName="maas-api"
Nov 02 02:34:43.361945 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.361841    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d45d7c5-9edd-465f-a034-7c483d43d47f" containerName="maas-api"
Nov 02 02:34:43.362007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.361983    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d45d7c5-9edd-465f-a034-7c483d43d47f" containerName="maas-api"
Nov 02 02:34:43.365473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.365452    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:43.407728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.407700    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363dc03a-7e8a-47ba-994e-5423819edf6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb\" (UID: \"363dc03a-7e8a-47ba-994e-5423819edf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:43.407853 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.407738    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wjx\" (UniqueName: \"kubernetes.io/projected/363dc03a-7e8a-47ba-994e-5423819edf6c-kube-api-access-66wjx\") pod \"payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb\" (UID: \"363dc03a-7e8a-47ba-994e-5423819edf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:43.419120 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.419091    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-71e54d09-dockercfg-kwccf\""
Nov 02 02:34:43.419403 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.419370    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-71e54d09\""
Nov 02 02:34:43.473784 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.473745    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"]
Nov 02 02:34:43.509038 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.509006    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363dc03a-7e8a-47ba-994e-5423819edf6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb\" (UID: \"363dc03a-7e8a-47ba-994e-5423819edf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:43.509264 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.509049    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wjx\" (UniqueName: \"kubernetes.io/projected/363dc03a-7e8a-47ba-994e-5423819edf6c-kube-api-access-66wjx\") pod \"payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb\" (UID: \"363dc03a-7e8a-47ba-994e-5423819edf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:43.509810 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.509789    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363dc03a-7e8a-47ba-994e-5423819edf6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb\" (UID: \"363dc03a-7e8a-47ba-994e-5423819edf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:43.514149 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.514122    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"]
Nov 02 02:34:43.518151 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.518134    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:43.570424 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.570397    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wjx\" (UniqueName: \"kubernetes.io/projected/363dc03a-7e8a-47ba-994e-5423819edf6c-kube-api-access-66wjx\") pod \"payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb\" (UID: \"363dc03a-7e8a-47ba-994e-5423819edf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:43.575824 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.575797    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"]
Nov 02 02:34:43.610773 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.610737    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375e9c40-023f-4a29-a002-2984ba9c3fff-plugins-config-volume\") pod \"payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k\" (UID: \"375e9c40-023f-4a29-a002-2984ba9c3fff\") " pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:43.610947 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.610884    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqk8\" (UniqueName: \"kubernetes.io/projected/375e9c40-023f-4a29-a002-2984ba9c3fff-kube-api-access-2nqk8\") pod \"payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k\" (UID: \"375e9c40-023f-4a29-a002-2984ba9c3fff\") " pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:43.679031 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.678996    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:43.712298 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.712266    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375e9c40-023f-4a29-a002-2984ba9c3fff-plugins-config-volume\") pod \"payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k\" (UID: \"375e9c40-023f-4a29-a002-2984ba9c3fff\") " pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:43.712449 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.712360    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqk8\" (UniqueName: \"kubernetes.io/projected/375e9c40-023f-4a29-a002-2984ba9c3fff-kube-api-access-2nqk8\") pod \"payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k\" (UID: \"375e9c40-023f-4a29-a002-2984ba9c3fff\") " pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:43.713028 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.713009    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375e9c40-023f-4a29-a002-2984ba9c3fff-plugins-config-volume\") pod \"payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k\" (UID: \"375e9c40-023f-4a29-a002-2984ba9c3fff\") " pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:43.773645 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.773615    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqk8\" (UniqueName: \"kubernetes.io/projected/375e9c40-023f-4a29-a002-2984ba9c3fff-kube-api-access-2nqk8\") pod \"payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k\" (UID: \"375e9c40-023f-4a29-a002-2984ba9c3fff\") " pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:43.831913 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:43.831878    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:44.055431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.055407    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"]
Nov 02 02:34:44.056669 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:34:44.056640    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363dc03a_7e8a_47ba_994e_5423819edf6c.slice/crio-1506fcd84f48600e8b8ad5c86e9a21717c115977d8cf79e6ba077244ea585dec WatchSource:0}: Error finding container 1506fcd84f48600e8b8ad5c86e9a21717c115977d8cf79e6ba077244ea585dec: Status 404 returned error can't find the container with id 1506fcd84f48600e8b8ad5c86e9a21717c115977d8cf79e6ba077244ea585dec
Nov 02 02:34:44.208622 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.208597    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"]
Nov 02 02:34:44.209775 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:34:44.209748    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375e9c40_023f_4a29_a002_2984ba9c3fff.slice/crio-51471ae213019f26cd1661a980a92c0c700641fd17a14afc32cafb9d60d086c3 WatchSource:0}: Error finding container 51471ae213019f26cd1661a980a92c0c700641fd17a14afc32cafb9d60d086c3: Status 404 returned error can't find the container with id 51471ae213019f26cd1661a980a92c0c700641fd17a14afc32cafb9d60d086c3
Nov 02 02:34:44.268836 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.268811    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"]
Nov 02 02:34:44.272714 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.272694    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.319470 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.319384    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d054858-3d0c-439c-88d8-7f874f917e7a-openshift-service-ca\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.319651 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.319525    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.319651 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.319636    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-metrics-certs\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.319775 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.319672    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/4d054858-3d0c-439c-88d8-7f874f917e7a-kube-api-access-7pdfp\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.327424 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.327401    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-71e54d09-serving-cert\""
Nov 02 02:34:44.331142 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.331119    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"]
Nov 02 02:34:44.420698 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.420648    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-metrics-certs\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.420886 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.420712    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/4d054858-3d0c-439c-88d8-7f874f917e7a-kube-api-access-7pdfp\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.420886 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.420835    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d054858-3d0c-439c-88d8-7f874f917e7a-openshift-service-ca\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.421007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.420913    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.421085 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:34:44.421044    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-serving-cert: secret "maas-api-e2e-ait-71e54d09-serving-cert" not found
Nov 02 02:34:44.421152 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:34:44.421128    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls podName:4d054858-3d0c-439c-88d8-7f874f917e7a nodeName:}" failed. No retries permitted until 2026-11-02 02:34:44.921107265 +0000 UTC m=+3777.826572616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls") pod "maas-api-e2e-ait-71e54d09-896d5d596-jvmck" (UID: "4d054858-3d0c-439c-88d8-7f874f917e7a") : secret "maas-api-e2e-ait-71e54d09-serving-cert" not found
Nov 02 02:34:44.421600 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.421576    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d054858-3d0c-439c-88d8-7f874f917e7a-openshift-service-ca\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.423714 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.423687    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-metrics-certs\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.484516 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.484482    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/4d054858-3d0c-439c-88d8-7f874f917e7a-kube-api-access-7pdfp\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.502222 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.502194    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:44.722797 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.722760    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:44.926764 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.926729    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:44.929349 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:44.929320    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls\") pod \"maas-api-e2e-ait-71e54d09-896d5d596-jvmck\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:45.013873 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.013779    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k" event={"ID":"375e9c40-023f-4a29-a002-2984ba9c3fff","Type":"ContainerStarted","Data":"786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6"}
Nov 02 02:34:45.013873 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.013847    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k" event={"ID":"375e9c40-023f-4a29-a002-2984ba9c3fff","Type":"ContainerStarted","Data":"51471ae213019f26cd1661a980a92c0c700641fd17a14afc32cafb9d60d086c3"}
Nov 02 02:34:45.014157 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.013877    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:45.015577 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.015550    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb" event={"ID":"363dc03a-7e8a-47ba-994e-5423819edf6c","Type":"ContainerStarted","Data":"41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719"}
Nov 02 02:34:45.015695 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.015593    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb" event={"ID":"363dc03a-7e8a-47ba-994e-5423819edf6c","Type":"ContainerStarted","Data":"1506fcd84f48600e8b8ad5c86e9a21717c115977d8cf79e6ba077244ea585dec"}
Nov 02 02:34:45.015695 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.015614    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:45.130147 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.130083    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k" podStartSLOduration=2.130040796 podStartE2EDuration="2.130040796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:34:45.128078601 +0000 UTC m=+3778.033543961" watchObservedRunningTime="2026-11-02 02:34:45.130040796 +0000 UTC m=+3778.035506152"
Nov 02 02:34:45.186945 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.186915    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:45.251258 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.251198    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb" podStartSLOduration=2.251172978 podStartE2EDuration="2.251172978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:34:45.244291354 +0000 UTC m=+3778.149756712" watchObservedRunningTime="2026-11-02 02:34:45.251172978 +0000 UTC m=+3778.156638361"
Nov 02 02:34:45.574790 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.574756    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"]
Nov 02 02:34:45.576401 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:34:45.576376    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d054858_3d0c_439c_88d8_7f874f917e7a.slice/crio-f026d6d6d41d07d4588f1d9bc3d1291c84f93060b1d7685f674b7151876951e9 WatchSource:0}: Error finding container f026d6d6d41d07d4588f1d9bc3d1291c84f93060b1d7685f674b7151876951e9: Status 404 returned error can't find the container with id f026d6d6d41d07d4588f1d9bc3d1291c84f93060b1d7685f674b7151876951e9
Nov 02 02:34:45.991608 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:45.991579    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-lfrzw"
Nov 02 02:34:46.022775 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:46.022738    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck" event={"ID":"4d054858-3d0c-439c-88d8-7f874f917e7a","Type":"ContainerStarted","Data":"f026d6d6d41d07d4588f1d9bc3d1291c84f93060b1d7685f674b7151876951e9"}
Nov 02 02:34:47.030193 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:47.030150    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck" event={"ID":"4d054858-3d0c-439c-88d8-7f874f917e7a","Type":"ContainerStarted","Data":"bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f"}
Nov 02 02:34:47.030678 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:47.030278    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:47.148295 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:47.148245    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck" podStartSLOduration=2.352788742 podStartE2EDuration="3.148230496s" totalImagesPullingTime="795.441754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:34:47.147300905 +0000 UTC m=+3780.052766263" watchObservedRunningTime="2026-11-02 02:34:47.148230496 +0000 UTC m=+3780.053695853"
Nov 02 02:34:49.939248 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:49.939211    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:53.043034 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:53.042997    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:34:54.874249 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:54.874213    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:56.026016 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:56.025983    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:34:56.028490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:56.026383    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:34:59.676970 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:59.676929    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:34:59.855772 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:34:59.855739    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:04.884325 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:04.884285    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:09.960202 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:09.960164    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:12.138386 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:35:12.138343    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:35:14.770419 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:14.770384    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:17.358490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:17.358460    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"]
Nov 02 02:35:17.461713 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:17.461679    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"]
Nov 02 02:35:17.756661 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:17.756622    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:17.952350 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:17.952300    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:19.666026 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:19.665987    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:22.359743 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.359697    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k" podUID="375e9c40-023f-4a29-a002-2984ba9c3fff" containerName="payload-processing" containerID="cri-o://786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6" gracePeriod=25
Nov 02 02:35:22.462465 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.462423    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb" podUID="363dc03a-7e8a-47ba-994e-5423819edf6c" containerName="payload-pre-processing" containerID="cri-o://41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719" gracePeriod=25
Nov 02 02:35:22.628580 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.628554    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:35:22.718844 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.718819    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:35:22.721029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.721009    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375e9c40-023f-4a29-a002-2984ba9c3fff-kube-api-access-2nqk8\" (UniqueName: \"kubernetes.io/projected/375e9c40-023f-4a29-a002-2984ba9c3fff-kube-api-access-2nqk8\") pod \"375e9c40-023f-4a29-a002-2984ba9c3fff\" (UID: \"375e9c40-023f-4a29-a002-2984ba9c3fff\") "
Nov 02 02:35:22.721121 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.721101    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375e9c40-023f-4a29-a002-2984ba9c3fff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375e9c40-023f-4a29-a002-2984ba9c3fff-plugins-config-volume\") pod \"375e9c40-023f-4a29-a002-2984ba9c3fff\" (UID: \"375e9c40-023f-4a29-a002-2984ba9c3fff\") "
Nov 02 02:35:22.721441 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.721419    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375e9c40-023f-4a29-a002-2984ba9c3fff-plugins-config-volume" pod "375e9c40-023f-4a29-a002-2984ba9c3fff" (UID: "375e9c40-023f-4a29-a002-2984ba9c3fff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:35:22.723279 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.723261    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375e9c40-023f-4a29-a002-2984ba9c3fff-kube-api-access-2nqk8" pod "375e9c40-023f-4a29-a002-2984ba9c3fff" (UID: "375e9c40-023f-4a29-a002-2984ba9c3fff"). InnerVolumeSpecName "kube-api-access-2nqk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:22.822288 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.822260    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363dc03a-7e8a-47ba-994e-5423819edf6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363dc03a-7e8a-47ba-994e-5423819edf6c-plugins-config-volume\") pod \"363dc03a-7e8a-47ba-994e-5423819edf6c\" (UID: \"363dc03a-7e8a-47ba-994e-5423819edf6c\") "
Nov 02 02:35:22.822465 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.822329    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363dc03a-7e8a-47ba-994e-5423819edf6c-kube-api-access-66wjx\" (UniqueName: \"kubernetes.io/projected/363dc03a-7e8a-47ba-994e-5423819edf6c-kube-api-access-66wjx\") pod \"363dc03a-7e8a-47ba-994e-5423819edf6c\" (UID: \"363dc03a-7e8a-47ba-994e-5423819edf6c\") "
Nov 02 02:35:22.822568 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.822553    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nqk8\" (UniqueName: \"kubernetes.io/projected/375e9c40-023f-4a29-a002-2984ba9c3fff-kube-api-access-2nqk8\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:22.822627 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.822571    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375e9c40-023f-4a29-a002-2984ba9c3fff-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:22.822743 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.822719    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363dc03a-7e8a-47ba-994e-5423819edf6c-plugins-config-volume" pod "363dc03a-7e8a-47ba-994e-5423819edf6c" (UID: "363dc03a-7e8a-47ba-994e-5423819edf6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:35:22.824578 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.824558    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363dc03a-7e8a-47ba-994e-5423819edf6c-kube-api-access-66wjx" pod "363dc03a-7e8a-47ba-994e-5423819edf6c" (UID: "363dc03a-7e8a-47ba-994e-5423819edf6c"). InnerVolumeSpecName "kube-api-access-66wjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:22.923756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.923720    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363dc03a-7e8a-47ba-994e-5423819edf6c-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:22.923756 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:22.923750    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wjx\" (UniqueName: \"kubernetes.io/projected/363dc03a-7e8a-47ba-994e-5423819edf6c-kube-api-access-66wjx\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:23.259435 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.259407    4912 generic.go:356] "Generic (PLEG): container finished" podID="363dc03a-7e8a-47ba-994e-5423819edf6c" containerID="41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719" exitCode=0
Nov 02 02:35:23.259607 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.259463    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"
Nov 02 02:35:23.259607 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.259499    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb" event={"ID":"363dc03a-7e8a-47ba-994e-5423819edf6c","Type":"ContainerDied","Data":"41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719"}
Nov 02 02:35:23.259607 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.259555    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb" event={"ID":"363dc03a-7e8a-47ba-994e-5423819edf6c","Type":"ContainerDied","Data":"1506fcd84f48600e8b8ad5c86e9a21717c115977d8cf79e6ba077244ea585dec"}
Nov 02 02:35:23.259607 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.259576    4912 scope.go:122] "RemoveContainer" containerID="41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719"
Nov 02 02:35:23.260988 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.260968    4912 generic.go:356] "Generic (PLEG): container finished" podID="375e9c40-023f-4a29-a002-2984ba9c3fff" containerID="786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6" exitCode=0
Nov 02 02:35:23.261123 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.261007    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k" event={"ID":"375e9c40-023f-4a29-a002-2984ba9c3fff","Type":"ContainerDied","Data":"786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6"}
Nov 02 02:35:23.261123 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.261037    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"
Nov 02 02:35:23.261123 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.261073    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k" event={"ID":"375e9c40-023f-4a29-a002-2984ba9c3fff","Type":"ContainerDied","Data":"51471ae213019f26cd1661a980a92c0c700641fd17a14afc32cafb9d60d086c3"}
Nov 02 02:35:23.270166 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.270145    4912 scope.go:122] "RemoveContainer" containerID="41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719"
Nov 02 02:35:23.270411 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:35:23.270393    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719\": container with ID starting with 41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719 not found: ID does not exist" containerID="41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719"
Nov 02 02:35:23.270468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.270418    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719"} err="failed to get container status \"41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719\": rpc error: code = NotFound desc = could not find container \"41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719\": container with ID starting with 41c4278a9231b2ef5b58595d211d43765f25766bf8065a983eded053e43ae719 not found: ID does not exist"
Nov 02 02:35:23.270468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.270437    4912 scope.go:122] "RemoveContainer" containerID="786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6"
Nov 02 02:35:23.278659 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.278644    4912 scope.go:122] "RemoveContainer" containerID="786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6"
Nov 02 02:35:23.278946 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:35:23.278928    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6\": container with ID starting with 786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6 not found: ID does not exist" containerID="786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6"
Nov 02 02:35:23.279011 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.278954    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6"} err="failed to get container status \"786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6\": rpc error: code = NotFound desc = could not find container \"786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6\": container with ID starting with 786c19b91fcb119ff622159d3e317be5b7b25025fc070e5a4fdc5ec2cfaf75c6 not found: ID does not exist"
Nov 02 02:35:23.424402 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.424365    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"]
Nov 02 02:35:23.473758 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.473729    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-71e54d09-579f6c7cc-kfb8k"]
Nov 02 02:35:23.541479 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.541401    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"]
Nov 02 02:35:23.547414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.547389    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-71e54d09-7f884cb66c-kjvdb"]
Nov 02 02:35:23.755407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.755370    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"]
Nov 02 02:35:23.755645 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.755608    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck" podUID="4d054858-3d0c-439c-88d8-7f874f917e7a" containerName="maas-api" containerID="cri-o://bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f" gracePeriod=30
Nov 02 02:35:23.879030 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:23.878994    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:24.028944 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.028916    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:35:24.136282 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.136190    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d054858-3d0c-439c-88d8-7f874f917e7a-kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/4d054858-3d0c-439c-88d8-7f874f917e7a-kube-api-access-7pdfp\") pod \"4d054858-3d0c-439c-88d8-7f874f917e7a\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") "
Nov 02 02:35:24.136696 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.136668    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls\") pod \"4d054858-3d0c-439c-88d8-7f874f917e7a\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") "
Nov 02 02:35:24.137211 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.137192    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d054858-3d0c-439c-88d8-7f874f917e7a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d054858-3d0c-439c-88d8-7f874f917e7a-openshift-service-ca\") pod \"4d054858-3d0c-439c-88d8-7f874f917e7a\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") "
Nov 02 02:35:24.137377 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.137236    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-metrics-certs\") pod \"4d054858-3d0c-439c-88d8-7f874f917e7a\" (UID: \"4d054858-3d0c-439c-88d8-7f874f917e7a\") "
Nov 02 02:35:24.138104 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.138078    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d054858-3d0c-439c-88d8-7f874f917e7a-openshift-service-ca" pod "4d054858-3d0c-439c-88d8-7f874f917e7a" (UID: "4d054858-3d0c-439c-88d8-7f874f917e7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:35:24.139420 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.139395    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d054858-3d0c-439c-88d8-7f874f917e7a-kube-api-access-7pdfp" pod "4d054858-3d0c-439c-88d8-7f874f917e7a" (UID: "4d054858-3d0c-439c-88d8-7f874f917e7a"). InnerVolumeSpecName "kube-api-access-7pdfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:24.140351 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.140323    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls" pod "4d054858-3d0c-439c-88d8-7f874f917e7a" (UID: "4d054858-3d0c-439c-88d8-7f874f917e7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:35:24.140440 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.140349    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-metrics-certs" pod "4d054858-3d0c-439c-88d8-7f874f917e7a" (UID: "4d054858-3d0c-439c-88d8-7f874f917e7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:35:24.239107 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.239073    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d054858-3d0c-439c-88d8-7f874f917e7a-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:24.239107 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.239104    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:24.239107 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.239113    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/4d054858-3d0c-439c-88d8-7f874f917e7a-kube-api-access-7pdfp\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:24.239366 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.239123    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d054858-3d0c-439c-88d8-7f874f917e7a-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:24.273954 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.273921    4912 generic.go:356] "Generic (PLEG): container finished" podID="4d054858-3d0c-439c-88d8-7f874f917e7a" containerID="bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f" exitCode=0
Nov 02 02:35:24.274186 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.273979    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"
Nov 02 02:35:24.274186 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.274001    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck" event={"ID":"4d054858-3d0c-439c-88d8-7f874f917e7a","Type":"ContainerDied","Data":"bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f"}
Nov 02 02:35:24.274186 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.274095    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck" event={"ID":"4d054858-3d0c-439c-88d8-7f874f917e7a","Type":"ContainerDied","Data":"f026d6d6d41d07d4588f1d9bc3d1291c84f93060b1d7685f674b7151876951e9"}
Nov 02 02:35:24.274186 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.274131    4912 scope.go:122] "RemoveContainer" containerID="bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f"
Nov 02 02:35:24.284961 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.284940    4912 scope.go:122] "RemoveContainer" containerID="bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f"
Nov 02 02:35:24.285292 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:35:24.285272    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f\": container with ID starting with bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f not found: ID does not exist" containerID="bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f"
Nov 02 02:35:24.285346 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.285301    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f"} err="failed to get container status \"bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f\": rpc error: code = NotFound desc = could not find container \"bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f\": container with ID starting with bfdc44b3d0dbc51fe949b75c95c67b1117c0d72b1435d8cec5b24481d919cd5f not found: ID does not exist"
Nov 02 02:35:24.437528 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.437441    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"]
Nov 02 02:35:24.441259 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:24.441231    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-71e54d09-896d5d596-jvmck"]
Nov 02 02:35:25.224919 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:25.224887    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363dc03a-7e8a-47ba-994e-5423819edf6c" path="/var/lib/kubelet/pods/363dc03a-7e8a-47ba-994e-5423819edf6c/volumes"
Nov 02 02:35:25.225324 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:25.225309    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375e9c40-023f-4a29-a002-2984ba9c3fff" path="/var/lib/kubelet/pods/375e9c40-023f-4a29-a002-2984ba9c3fff/volumes"
Nov 02 02:35:25.225664 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:25.225652    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d054858-3d0c-439c-88d8-7f874f917e7a" path="/var/lib/kubelet/pods/4d054858-3d0c-439c-88d8-7f874f917e7a/volumes"
Nov 02 02:35:25.411739 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:25.411699    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:25.807759 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:25.807719    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:26.856468 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:26.856431    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:28.115334 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:28.115295    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:28.747588 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:28.747549    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:29.053619 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:29.053531    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:32.782116 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:32.782074    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:32.975352 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:32.975311    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:33.159427 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:33.159387    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:35.466844 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.466806    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"]
Nov 02 02:35:35.467534 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.467512    4912 cpu_manager.go:414] "containerMap: removing container" podUID="363dc03a-7e8a-47ba-994e-5423819edf6c" containerName="payload-pre-processing"
Nov 02 02:35:35.467649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.467535    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363dc03a-7e8a-47ba-994e-5423819edf6c" containerName="payload-pre-processing"
Nov 02 02:35:35.467649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.467549    4912 cpu_manager.go:414] "containerMap: removing container" podUID="4d054858-3d0c-439c-88d8-7f874f917e7a" containerName="maas-api"
Nov 02 02:35:35.467649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.467556    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d054858-3d0c-439c-88d8-7f874f917e7a" containerName="maas-api"
Nov 02 02:35:35.467649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.467583    4912 cpu_manager.go:414] "containerMap: removing container" podUID="375e9c40-023f-4a29-a002-2984ba9c3fff" containerName="payload-processing"
Nov 02 02:35:35.467649 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.467590    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375e9c40-023f-4a29-a002-2984ba9c3fff" containerName="payload-processing"
Nov 02 02:35:35.467910 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.467736    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="363dc03a-7e8a-47ba-994e-5423819edf6c" containerName="payload-pre-processing"
Nov 02 02:35:35.467910 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.467752    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d054858-3d0c-439c-88d8-7f874f917e7a" containerName="maas-api"
Nov 02 02:35:35.467910 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.467763    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="375e9c40-023f-4a29-a002-2984ba9c3fff" containerName="payload-processing"
Nov 02 02:35:35.472589 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.472567    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.549880 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.549833    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.550082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.550001    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-metrics-certs\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.550143 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.550103    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhkb\" (UniqueName: \"kubernetes.io/projected/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-kube-api-access-8jhkb\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.550194 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.550166    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.574963 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.574932    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-433aff-serving-cert\""
Nov 02 02:35:35.578002 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.577973    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"]
Nov 02 02:35:35.651761 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.651732    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.652000 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.651790    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-metrics-certs\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.652000 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.651808    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhkb\" (UniqueName: \"kubernetes.io/projected/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-kube-api-access-8jhkb\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.652000 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.651844    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.652180 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:35:35.652005    4912 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-serving-cert: secret "maas-api-e2e-authcfg-http-433aff-serving-cert" not found
Nov 02 02:35:35.652180 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:35:35.652103    4912 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls podName:90c2ed20-1036-4bf6-aec9-4169c2c2c6a0 nodeName:}" failed. No retries permitted until 2026-11-02 02:35:36.152081047 +0000 UTC m=+3829.057546396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls") pod "maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd" (UID: "90c2ed20-1036-4bf6-aec9-4169c2c2c6a0") : secret "maas-api-e2e-authcfg-http-433aff-serving-cert" not found
Nov 02 02:35:35.652595 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.652572    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.654493 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.654459    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-metrics-certs\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.700691 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.700649    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:35.714431 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.714394    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhkb\" (UniqueName: \"kubernetes.io/projected/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-kube-api-access-8jhkb\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:35.890848 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:35.890815    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:36.082225 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.082184    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:36.157375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.157289    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:36.159925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.159897    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:36.259852 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.259812    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:36.385783 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.385756    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:36.734829 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.734753    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"]
Nov 02 02:35:36.739757 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.739738    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:36.767255 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:35:36.767225    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c2ed20_1036_4bf6_aec9_4169c2c2c6a0.slice/crio-47a2bc4fd596a7c57dfa3afdda1a75094dbb639902f78632346d1106da2bf9fb WatchSource:0}: Error finding container 47a2bc4fd596a7c57dfa3afdda1a75094dbb639902f78632346d1106da2bf9fb: Status 404 returned error can't find the container with id 47a2bc4fd596a7c57dfa3afdda1a75094dbb639902f78632346d1106da2bf9fb
Nov 02 02:35:36.784603 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.784571    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"]
Nov 02 02:35:36.812542 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.812516    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-433aff\""
Nov 02 02:35:36.812717 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.812596    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-433aff-dockercfg-ch4lf\""
Nov 02 02:35:36.815683 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.815659    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"]
Nov 02 02:35:36.866123 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.866087    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300bae68-bc4d-4ab4-be05-024ec6f54867-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms\" (UID: \"300bae68-bc4d-4ab4-be05-024ec6f54867\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:36.866297 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.866187    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lqgc\" (UniqueName: \"kubernetes.io/projected/300bae68-bc4d-4ab4-be05-024ec6f54867-kube-api-access-7lqgc\") pod \"payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms\" (UID: \"300bae68-bc4d-4ab4-be05-024ec6f54867\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:36.966883 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.966848    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300bae68-bc4d-4ab4-be05-024ec6f54867-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms\" (UID: \"300bae68-bc4d-4ab4-be05-024ec6f54867\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:36.967122 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.966993    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqgc\" (UniqueName: \"kubernetes.io/projected/300bae68-bc4d-4ab4-be05-024ec6f54867-kube-api-access-7lqgc\") pod \"payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms\" (UID: \"300bae68-bc4d-4ab4-be05-024ec6f54867\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:36.967579 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:36.967560    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300bae68-bc4d-4ab4-be05-024ec6f54867-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms\" (UID: \"300bae68-bc4d-4ab4-be05-024ec6f54867\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:37.028808 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:37.028735    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lqgc\" (UniqueName: \"kubernetes.io/projected/300bae68-bc4d-4ab4-be05-024ec6f54867-kube-api-access-7lqgc\") pod \"payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms\" (UID: \"300bae68-bc4d-4ab4-be05-024ec6f54867\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:37.054371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:37.054337    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:37.363289 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:37.363204    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd" event={"ID":"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0","Type":"ContainerStarted","Data":"47a2bc4fd596a7c57dfa3afdda1a75094dbb639902f78632346d1106da2bf9fb"}
Nov 02 02:35:37.432938 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:37.432909    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"]
Nov 02 02:35:37.434483 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:35:37.434455    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300bae68_bc4d_4ab4_be05_024ec6f54867.slice/crio-384c5d152b8b2d714386177b99af9173cb0df40023b8a10fd94692c7bae49d5f WatchSource:0}: Error finding container 384c5d152b8b2d714386177b99af9173cb0df40023b8a10fd94692c7bae49d5f: Status 404 returned error can't find the container with id 384c5d152b8b2d714386177b99af9173cb0df40023b8a10fd94692c7bae49d5f
Nov 02 02:35:38.371087 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:38.371020    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd" event={"ID":"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0","Type":"ContainerStarted","Data":"36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c"}
Nov 02 02:35:38.371546 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:38.371137    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:38.372677 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:38.372655    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms" event={"ID":"300bae68-bc4d-4ab4-be05-024ec6f54867","Type":"ContainerStarted","Data":"da8f1cdad23b780067f1d1a83d391753db8aadfae7dcfb453ac048b312ee1252"}
Nov 02 02:35:38.372738 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:38.372695    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms" event={"ID":"300bae68-bc4d-4ab4-be05-024ec6f54867","Type":"ContainerStarted","Data":"384c5d152b8b2d714386177b99af9173cb0df40023b8a10fd94692c7bae49d5f"}
Nov 02 02:35:38.372798 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:38.372785    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:38.494036 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:38.493975    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd" podStartSLOduration=2.6258629730000003 podStartE2EDuration="3.493960963s" totalImagesPullingTime="868.09799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:35:38.490813769 +0000 UTC m=+3831.396279126" watchObservedRunningTime="2026-11-02 02:35:38.493960963 +0000 UTC m=+3831.399426320"
Nov 02 02:35:38.610460 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:38.610408    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms" podStartSLOduration=2.610393207 podStartE2EDuration="2.610393207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:35:38.607450291 +0000 UTC m=+3831.512915667" watchObservedRunningTime="2026-11-02 02:35:38.610393207 +0000 UTC m=+3831.515858564"
Nov 02 02:35:40.754976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:40.754943    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:42.139573 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:35:42.139533    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:35:44.385708 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:44.385674    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:35:45.681519 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:45.681490    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:49.382967 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:49.382930    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:35:50.144902 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:50.144869    4912 scope.go:122] "RemoveContainer" containerID="8231e99598752ce4c0c1a4401f139587c146bd19d1bf24bf817ff74e40e564bd"
Nov 02 02:35:50.659277 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:50.659230    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:52.486775 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:52.486729    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:35:55.952029 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:35:55.951992    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:00.128105 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:00.128046    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:00.778267 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:00.778225    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:05.787488 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:05.787447    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:10.703010 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:10.702975    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:12.137613 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:36:12.137568    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:36:15.668026 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:15.667972    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:20.456758 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:20.456713    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:25.473781 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:25.473738    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:30.512759 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:30.512724    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:31.414078 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:31.414025    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:35.747148 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:35.747101    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:40.809959 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:40.809917    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:42.138461 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:36:42.138425    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:36:43.431567 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:43.431517    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:45.861399 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:45.861323    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:48.023943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:48.023904    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:36:48.030477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:48.030456    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:36:48.179834 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:48.179808    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m8ck_277a3a65-7e29-4cb7-a913-d364040935b0/console-operator/3.log"
Nov 02 02:36:48.186675 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:48.186649    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxgd2_5df2f3d5-e8eb-48e4-b196-2c08c1fd28ce/ovn-acl-logging/0.log"
Nov 02 02:36:51.015979 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:51.015938    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:36:55.972724 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:36:55.972689    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:00.491355 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:00.491275    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:01.022180 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:01.022138    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:06.055847 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:06.055808    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:11.023726 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:11.023689    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:12.137949 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:37:12.137905    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:37:16.320593 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:16.320549    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:18.541876 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:18.541838    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:21.384473 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:21.384378    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:24.883001 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:24.882962    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"]
Nov 02 02:37:25.143732 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:25.143635    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:25.414838 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:25.414720    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:29.883609 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:29.883570    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms" podUID="300bae68-bc4d-4ab4-be05-024ec6f54867" containerName="payload-pre-processing" containerID="cri-o://da8f1cdad23b780067f1d1a83d391753db8aadfae7dcfb453ac048b312ee1252" gracePeriod=25
Nov 02 02:37:30.096355 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:30.092663    4912 generic.go:356] "Generic (PLEG): container finished" podID="300bae68-bc4d-4ab4-be05-024ec6f54867" containerID="da8f1cdad23b780067f1d1a83d391753db8aadfae7dcfb453ac048b312ee1252" exitCode=0
Nov 02 02:37:30.096355 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:30.092978    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms" event={"ID":"300bae68-bc4d-4ab4-be05-024ec6f54867","Type":"ContainerDied","Data":"da8f1cdad23b780067f1d1a83d391753db8aadfae7dcfb453ac048b312ee1252"}
Nov 02 02:37:30.155283 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:30.155251    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:37:30.197591 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:30.197554    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300bae68-bc4d-4ab4-be05-024ec6f54867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300bae68-bc4d-4ab4-be05-024ec6f54867-plugins-config-volume\") pod \"300bae68-bc4d-4ab4-be05-024ec6f54867\" (UID: \"300bae68-bc4d-4ab4-be05-024ec6f54867\") "
Nov 02 02:37:30.197779 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:30.197693    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300bae68-bc4d-4ab4-be05-024ec6f54867-kube-api-access-7lqgc\" (UniqueName: \"kubernetes.io/projected/300bae68-bc4d-4ab4-be05-024ec6f54867-kube-api-access-7lqgc\") pod \"300bae68-bc4d-4ab4-be05-024ec6f54867\" (UID: \"300bae68-bc4d-4ab4-be05-024ec6f54867\") "
Nov 02 02:37:30.197943 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:30.197916    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300bae68-bc4d-4ab4-be05-024ec6f54867-plugins-config-volume" pod "300bae68-bc4d-4ab4-be05-024ec6f54867" (UID: "300bae68-bc4d-4ab4-be05-024ec6f54867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:37:30.198093 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:30.198045    4912 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300bae68-bc4d-4ab4-be05-024ec6f54867-plugins-config-volume\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:37:30.200066 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:30.200032    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300bae68-bc4d-4ab4-be05-024ec6f54867-kube-api-access-7lqgc" pod "300bae68-bc4d-4ab4-be05-024ec6f54867" (UID: "300bae68-bc4d-4ab4-be05-024ec6f54867"). InnerVolumeSpecName "kube-api-access-7lqgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:37:30.299877 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:30.299836    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lqgc\" (UniqueName: \"kubernetes.io/projected/300bae68-bc4d-4ab4-be05-024ec6f54867-kube-api-access-7lqgc\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:37:31.101932 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:31.101906    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"
Nov 02 02:37:31.102414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:31.101937    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms" event={"ID":"300bae68-bc4d-4ab4-be05-024ec6f54867","Type":"ContainerDied","Data":"384c5d152b8b2d714386177b99af9173cb0df40023b8a10fd94692c7bae49d5f"}
Nov 02 02:37:31.102414 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:31.101988    4912 scope.go:122] "RemoveContainer" containerID="da8f1cdad23b780067f1d1a83d391753db8aadfae7dcfb453ac048b312ee1252"
Nov 02 02:37:31.268132 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:31.268097    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"]
Nov 02 02:37:31.317816 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:31.317786    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-433aff-65dfbf4579-9bwms"]
Nov 02 02:37:33.227960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:33.227925    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300bae68-bc4d-4ab4-be05-024ec6f54867" path="/var/lib/kubelet/pods/300bae68-bc4d-4ab4-be05-024ec6f54867/volumes"
Nov 02 02:37:35.415550 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.415512    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"]
Nov 02 02:37:35.416040 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.415737    4912 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd" podUID="90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" containerName="maas-api" containerID="cri-o://36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c" gracePeriod=30
Nov 02 02:37:35.607465 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.607417    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:35.687843 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.687816    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:37:35.756226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.756193    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-kube-api-access-8jhkb\" (UniqueName: \"kubernetes.io/projected/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-kube-api-access-8jhkb\") pod \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") "
Nov 02 02:37:35.756226 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.756228    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls\") pod \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") "
Nov 02 02:37:35.756503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.756266    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-openshift-service-ca\") pod \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") "
Nov 02 02:37:35.756503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.756349    4912 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-metrics-certs\") pod \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\" (UID: \"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0\") "
Nov 02 02:37:35.756719 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.756686    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-openshift-service-ca" pod "90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" (UID: "90c2ed20-1036-4bf6-aec9-4169c2c2c6a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:37:35.758698 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.758667    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-metrics-certs" pod "90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" (UID: "90c2ed20-1036-4bf6-aec9-4169c2c2c6a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:37:35.758805 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.758746    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-kube-api-access-8jhkb" pod "90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" (UID: "90c2ed20-1036-4bf6-aec9-4169c2c2c6a0"). InnerVolumeSpecName "kube-api-access-8jhkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:37:35.758805 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.758743    4912 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls" pod "90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" (UID: "90c2ed20-1036-4bf6-aec9-4169c2c2c6a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:37:35.858246 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.858205    4912 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-metrics-certs\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:37:35.858246 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.858241    4912 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jhkb\" (UniqueName: \"kubernetes.io/projected/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-kube-api-access-8jhkb\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:37:35.858472 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.858257    4912 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-maas-api-tls\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:37:35.858472 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:35.858270    4912 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0-openshift-service-ca\") on node \"ip-10-0-135-33.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:37:36.140234 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:36.140195    4912 generic.go:356] "Generic (PLEG): container finished" podID="90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" containerID="36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c" exitCode=0
Nov 02 02:37:36.140433 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:36.140243    4912 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"
Nov 02 02:37:36.140433 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:36.140281    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd" event={"ID":"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0","Type":"ContainerDied","Data":"36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c"}
Nov 02 02:37:36.140433 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:36.140340    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd" event={"ID":"90c2ed20-1036-4bf6-aec9-4169c2c2c6a0","Type":"ContainerDied","Data":"47a2bc4fd596a7c57dfa3afdda1a75094dbb639902f78632346d1106da2bf9fb"}
Nov 02 02:37:36.140433 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:36.140370    4912 scope.go:122] "RemoveContainer" containerID="36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c"
Nov 02 02:37:36.151289 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:36.151264    4912 scope.go:122] "RemoveContainer" containerID="36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c"
Nov 02 02:37:36.151601 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:37:36.151577    4912 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c\": container with ID starting with 36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c not found: ID does not exist" containerID="36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c"
Nov 02 02:37:36.151652 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:36.151612    4912 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c"} err="failed to get container status \"36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c\": rpc error: code = NotFound desc = could not find container \"36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c\": container with ID starting with 36754714bb167c5d5179a519e09944e9c4ac7b9650937e4526db448603b90e9c not found: ID does not exist"
Nov 02 02:37:36.315247 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:36.315213    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"]
Nov 02 02:37:36.368498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:36.368457    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-433aff-64f8f8d5bf-wsftd"]
Nov 02 02:37:37.148537 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:37.148503    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:37.225007 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:37.224971    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" path="/var/lib/kubelet/pods/90c2ed20-1036-4bf6-aec9-4169c2c2c6a0/volumes"
Nov 02 02:37:37.448976 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:37.448886    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:38.365088 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:38.365039    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:42.137428 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:37:42.137389    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:37:44.855758 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:44.855727    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:45.125968 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:45.125922    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:45.353179 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:45.353139    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:55.725892 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:55.725756    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:37:56.855213 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:56.855166    4912 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-fhsck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:37:56.855839 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:37:56.855802    4912 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-fhsck" podUID="da8141a6-dfcd-4672-a751-1d2f60d39bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:38:00.866622 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:38:00.866589    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:38:12.139271 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:38:12.139232    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:38:28.476202 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:38:28.476111    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:38:29.047713 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:38:29.047674    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"]
Nov 02 02:38:29.098545 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:38:29.098504    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9r7x"]
Nov 02 02:38:29.224925 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:38:29.224891    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55a8923-fa38-4667-aa16-0e42f6b12644" path="/var/lib/kubelet/pods/a55a8923-fa38-4667-aa16-0e42f6b12644/volumes"
Nov 02 02:38:29.683082 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:38:29.683019    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:38:42.137425 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:38:42.137382    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:38:53.047385 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:38:53.047341    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"]
Nov 02 02:38:53.096561 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:38:53.096519    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-srwcn"]
Nov 02 02:38:53.225322 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:38:53.225289    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e256dff-e6e1-4589-8d02-12692ff79b61" path="/var/lib/kubelet/pods/6e256dff-e6e1-4589-8d02-12692ff79b61/volumes"
Nov 02 02:39:00.365118 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:00.365084    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd76fddf7-gwxbn_d7f049e4-ddcb-45e1-8819-2919806f1f3b/authorino/0.log"
Nov 02 02:39:01.166512 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:01.166473    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:39:11.816407 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:11.816376    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dpzkp_c0f61dd1-d6b3-4ac6-8f79-2a241265c803/manager/0.log"
Nov 02 02:39:12.103564 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:12.103486    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xgnbh_34c90cfc-6bba-478f-b339-afff5290ffa0/manager/0.log"
Nov 02 02:39:12.137607 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:39:12.137572    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:39:13.740442 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:13.740388    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-q49dp_dc1dccb6-466f-4fba-85fb-528e41a92d87/server/0.log"
Nov 02 02:39:14.382543 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:14.382514    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7bpsx_781242d4-1321-4bb2-8c4d-e27af6cc95c6/proxy/0.log"
Nov 02 02:39:15.567926 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:15.567887    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d4c89cd86-lfrzw_439c3d1f-470d-4ec6-a72d-13c031097a64/maas-api/0.log"
Nov 02 02:39:16.303737 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:16.303703    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b181ed9a-2t85f_ad197f27-bbbd-4f75-a89c-71ccc59343b2/revoke-keys/0.log"
Nov 02 02:39:16.731912 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:16.731880    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-6vlcv_4e728077-b906-4311-a8d3-15ebb7d56238/postgres/0.log"
Nov 02 02:39:18.294097 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:18.294041    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd76fddf7-gwxbn_d7f049e4-ddcb-45e1-8819-2919806f1f3b/authorino/0.log"
Nov 02 02:39:18.752181 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:18.752145    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-2hxrh_0754beef-39a5-42f3-affc-ba64da2c458d/manager/0.log"
Nov 02 02:39:18.975842 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:18.975808    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-j5pnp_f2037c89-aaf1-459d-9676-593d32c0ce50/manager/0.log"
Nov 02 02:39:19.048112 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:19.048034    4912 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"]
Nov 02 02:39:19.097388 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:19.097360    4912 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181ed9a-2t85f"]
Nov 02 02:39:19.225170 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:19.225136    4912 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad197f27-bbbd-4f75-a89c-71ccc59343b2" path="/var/lib/kubelet/pods/ad197f27-bbbd-4f75-a89c-71ccc59343b2/volumes"
Nov 02 02:39:19.461623 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:19.461592    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dtvzz_fc76523a-ee72-4475-b966-2fbde9c0d0b5/manager/0.log"
Nov 02 02:39:20.055700 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:20.055668    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pvcfb_2a3f2067-4ebd-4f7b-92c6-89715e73ac90/manager/0.log"
Nov 02 02:39:20.868103 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:20.868047    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485vxgl6_4af3acb7-dc87-454d-b1de-70d65476d689/istio-proxy/0.log"
Nov 02 02:39:21.923734 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:21.923703    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-4tqpd_ea71fe46-235c-43bb-adc6-708430a55abb/istio-proxy/0.log"
Nov 02 02:39:22.845126 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:22.845089    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-c7sz5_1fa36757-ccb1-4e9d-af8b-2a7ad54b7178/istio-proxy/0.log"
Nov 02 02:39:23.122564 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:23.122527    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-fhsck_da8141a6-dfcd-4672-a751-1d2f60d39bed/istio-proxy/0.log"
Nov 02 02:39:24.937251 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:24.937218    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk_a58eea3b-0fb7-4fd8-98ae-415f4972c7cf/istio-proxy/0.log"
Nov 02 02:39:25.212892 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:25.212799    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dtmjd_a7d724fe-3d4a-493e-9783-3ad4f3d0f500/payload-pre-processing/0.log"
Nov 02 02:39:25.422577 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:25.422552    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fv5bz_a2bd54dd-f5ec-4a45-b1fc-8534899e9b64/payload-processing/0.log"
Nov 02 02:39:25.652530 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:25.652502    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7948cc5b95-b87qg_5826852c-0e9b-4031-8213-94051d31c42c/router/0.log"
Nov 02 02:39:26.478265 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:26.478234    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p_94358ff5-ab4d-4e82-880f-515d87a4a790/main/0.log"
Nov 02 02:39:27.004728 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:27.004697    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-pgx5n_825d5fda-b233-458a-93f7-31f99cecadb3/main/0.log"
Nov 02 02:39:27.873222 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:27.873188    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-46j9s_8df16c8c-28c7-46e5-a791-0bed28ecb05a/main/0.log"
Nov 02 02:39:28.158089 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:28.158003    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg_62cc99ac-a5bc-4c99-a2e6-8487bb9f994e/main/0.log"
Nov 02 02:39:29.418376 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:29.418271    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd76fddf7-gwxbn_d7f049e4-ddcb-45e1-8819-2919806f1f3b/authorino/0.log"
Nov 02 02:39:38.839971 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:38.839932    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dpzkp_c0f61dd1-d6b3-4ac6-8f79-2a241265c803/manager/0.log"
Nov 02 02:39:39.112041 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:39.111966    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xgnbh_34c90cfc-6bba-478f-b339-afff5290ffa0/manager/0.log"
Nov 02 02:39:40.259041 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:40.259009    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-q49dp_dc1dccb6-466f-4fba-85fb-528e41a92d87/server/0.log"
Nov 02 02:39:40.860002 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:40.859968    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7bpsx_781242d4-1321-4bb2-8c4d-e27af6cc95c6/proxy/0.log"
Nov 02 02:39:41.813837 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:41.813640    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d4c89cd86-lfrzw_439c3d1f-470d-4ec6-a72d-13c031097a64/maas-api/0.log"
Nov 02 02:39:42.137155 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:39:42.137110    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:39:42.782527 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:42.782492    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-6vlcv_4e728077-b906-4311-a8d3-15ebb7d56238/postgres/0.log"
Nov 02 02:39:43.963459 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:43.963430    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd76fddf7-gwxbn_d7f049e4-ddcb-45e1-8819-2919806f1f3b/authorino/0.log"
Nov 02 02:39:44.434659 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:44.434632    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-2hxrh_0754beef-39a5-42f3-affc-ba64da2c458d/manager/0.log"
Nov 02 02:39:44.659789 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:44.659766    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-j5pnp_f2037c89-aaf1-459d-9676-593d32c0ce50/manager/0.log"
Nov 02 02:39:45.090164 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:45.090133    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dtvzz_fc76523a-ee72-4475-b966-2fbde9c0d0b5/manager/0.log"
Nov 02 02:39:45.583835 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:45.583803    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pvcfb_2a3f2067-4ebd-4f7b-92c6-89715e73ac90/manager/0.log"
Nov 02 02:39:46.191651 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:46.191616    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485vxgl6_4af3acb7-dc87-454d-b1de-70d65476d689/istio-proxy/0.log"
Nov 02 02:39:47.209375 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:47.209339    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-4tqpd_ea71fe46-235c-43bb-adc6-708430a55abb/istio-proxy/0.log"
Nov 02 02:39:48.086372 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:48.086342    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-c7sz5_1fa36757-ccb1-4e9d-af8b-2a7ad54b7178/istio-proxy/0.log"
Nov 02 02:39:48.371847 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:48.371813    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-fhsck_da8141a6-dfcd-4672-a751-1d2f60d39bed/istio-proxy/0.log"
Nov 02 02:39:50.214205 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:50.214171    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-5x2hk_a58eea3b-0fb7-4fd8-98ae-415f4972c7cf/istio-proxy/0.log"
Nov 02 02:39:50.224795 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:50.224767    4912 scope.go:122] "RemoveContainer" containerID="f4e81b9ee6f71e040d49b4cd2f7993c1ff4aa2fe745766e1c8956a9fd7c0c60c"
Nov 02 02:39:50.234553 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:50.234531    4912 scope.go:122] "RemoveContainer" containerID="b44569f06f7e21e20fc5e00719095fcafa15481bd6249533f6131b112768696c"
Nov 02 02:39:50.483165 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:50.483079    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dtmjd_a7d724fe-3d4a-493e-9783-3ad4f3d0f500/payload-pre-processing/0.log"
Nov 02 02:39:50.700371 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:50.700342    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fv5bz_a2bd54dd-f5ec-4a45-b1fc-8534899e9b64/payload-processing/0.log"
Nov 02 02:39:50.923648 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:50.923613    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7948cc5b95-b87qg_5826852c-0e9b-4031-8213-94051d31c42c/router/0.log"
Nov 02 02:39:51.533490 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:51.533459    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hhh8p_94358ff5-ab4d-4e82-880f-515d87a4a790/main/0.log"
Nov 02 02:39:52.069484 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:52.069452    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-pgx5n_825d5fda-b233-458a-93f7-31f99cecadb3/main/0.log"
Nov 02 02:39:52.869008 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:52.868975    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-46j9s_8df16c8c-28c7-46e5-a791-0bed28ecb05a/main/0.log"
Nov 02 02:39:53.147393 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:39:53.147310    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ww5vg_62cc99ac-a5bc-4c99-a2e6-8487bb9f994e/main/0.log"
Nov 02 02:40:01.527450 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:40:01.527370    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:40:12.138261 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:40:12.138221    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:40:22.747906 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:40:22.747876    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fv5bz_a2bd54dd-f5ec-4a45-b1fc-8534899e9b64/payload-processing/0.log"
Nov 02 02:40:22.975742 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:40:22.975713    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fv5bz_a2bd54dd-f5ec-4a45-b1fc-8534899e9b64/payload-processing/0.log"
Nov 02 02:40:42.138082 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:40:42.138027    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:40:50.262074 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:40:50.262038    4912 scope.go:122] "RemoveContainer" containerID="babbb793691b4372b1a5a0554032f9f0389e082cb413c312de8f4f4d1c0b36bd"
Nov 02 02:40:51.410584 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:40:51.410550    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:40:51.639706 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:40:51.639669    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:40:59.774102 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:40:59.774074    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fv5bz_a2bd54dd-f5ec-4a45-b1fc-8534899e9b64/payload-processing/0.log"
Nov 02 02:40:59.999903 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:40:59.999876    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fv5bz_a2bd54dd-f5ec-4a45-b1fc-8534899e9b64/payload-processing/0.log"
Nov 02 02:41:00.400467 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:00.400432    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:41:00.706677 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:00.706594    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:41:01.879523 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:01.879464    4912 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log" limit=1024
Nov 02 02:41:10.376310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:10.376275    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nfs6f_447b6716-29a3-4000-8df4-022f92de0e23/global-pull-secret-syncer/0.log"
Nov 02 02:41:10.969896 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:10.969869    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-57lbv_f74b04ce-df4a-46d3-9eb5-52464d2b5e50/connection-checker/0.log"
Nov 02 02:41:11.389178 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:11.389154    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5tvxk_444113a1-c287-4a62-98ba-7b88fe123732/connection-checker/0.log"
Nov 02 02:41:11.809155 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:11.809074    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xkcjd_230459c8-baa4-47cc-9cc2-73f0535fb650/connection-checker/0.log"
Nov 02 02:41:12.137883 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:41:12.137850    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"
Nov 02 02:41:12.295733 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:12.295700    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f2jxf_2aba37fc-45ec-4861-b481-f022d9727476/konnectivity-agent/0.log"
Nov 02 02:41:13.733320 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:13.733280    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-33.us-west-1.compute.internal_de43b58832dec4b7f2ea5d9fc15899cf/haproxy/0.log"
Nov 02 02:41:18.678361 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:18.678313    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd76fddf7-gwxbn_d7f049e4-ddcb-45e1-8819-2919806f1f3b/authorino/0.log"
Nov 02 02:41:19.007503 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.007409    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-2hxrh_0754beef-39a5-42f3-affc-ba64da2c458d/manager/0.log"
Nov 02 02:41:19.124801 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.124767    4912 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"]
Nov 02 02:41:19.125310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.125294    4912 cpu_manager.go:414] "containerMap: removing container" podUID="300bae68-bc4d-4ab4-be05-024ec6f54867" containerName="payload-pre-processing"
Nov 02 02:41:19.125310 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.125310    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300bae68-bc4d-4ab4-be05-024ec6f54867" containerName="payload-pre-processing"
Nov 02 02:41:19.125498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.125324    4912 cpu_manager.go:414] "containerMap: removing container" podUID="90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" containerName="maas-api"
Nov 02 02:41:19.125498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.125329    4912 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" containerName="maas-api"
Nov 02 02:41:19.125498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.125398    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="300bae68-bc4d-4ab4-be05-024ec6f54867" containerName="payload-pre-processing"
Nov 02 02:41:19.125498 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.125409    4912 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c2ed20-1036-4bf6-aec9-4169c2c2c6a0" containerName="maas-api"
Nov 02 02:41:19.128675 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.128660    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.182225 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.182180    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfrs6\"/\"openshift-service-ca.crt\""
Nov 02 02:41:19.226525 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.226495    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-podres\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.226679 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.226546    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-lib-modules\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.226679 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.226619    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqpg\" (UniqueName: \"kubernetes.io/projected/c404169e-f460-414e-8dee-defda48040e1-kube-api-access-drqpg\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.226679 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.226663    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-sys\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.226679 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.226677    4912 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-proc\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.231633 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.231605    4912 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfrs6\"/\"kube-root-ca.crt\""
Nov 02 02:41:19.231814 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.231786    4912 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cfrs6\"/\"default-dockercfg-ppkg9\""
Nov 02 02:41:19.234703 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.234684    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"]
Nov 02 02:41:19.328201 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.328115    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-lib-modules\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.328201 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.328171    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqpg\" (UniqueName: \"kubernetes.io/projected/c404169e-f460-414e-8dee-defda48040e1-kube-api-access-drqpg\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.328477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.328218    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-sys\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.328477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.328232    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-proc\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.328477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.328298    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-sys\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.328477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.328298    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-lib-modules\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.328477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.328332    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-proc\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.328477 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.328404    4912 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-podres\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.328708 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.328491    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c404169e-f460-414e-8dee-defda48040e1-podres\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.389162 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.389129    4912 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqpg\" (UniqueName: \"kubernetes.io/projected/c404169e-f460-414e-8dee-defda48040e1-kube-api-access-drqpg\") pod \"perf-node-gather-daemonset-jd2rl\" (UID: \"c404169e-f460-414e-8dee-defda48040e1\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.421978 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.421947    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-j5pnp_f2037c89-aaf1-459d-9676-593d32c0ce50/manager/0.log"
Nov 02 02:41:19.441445 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.441423    4912 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:19.815859 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:19.815829    4912 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"]
Nov 02 02:41:19.817117 ip-10-0-135-33 kubenswrapper[4912]: W1102 02:41:19.817088    4912 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc404169e_f460_414e_8dee_defda48040e1.slice/crio-a9952003981c3ae11c02e625adce39f1d82a27abf91371b5ddc494c6fa9aa699 WatchSource:0}: Error finding container a9952003981c3ae11c02e625adce39f1d82a27abf91371b5ddc494c6fa9aa699: Status 404 returned error can't find the container with id a9952003981c3ae11c02e625adce39f1d82a27abf91371b5ddc494c6fa9aa699
Nov 02 02:41:20.293323 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:20.293295    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dtvzz_fc76523a-ee72-4475-b966-2fbde9c0d0b5/manager/0.log"
Nov 02 02:41:20.650770 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:20.650735    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl" event={"ID":"c404169e-f460-414e-8dee-defda48040e1","Type":"ContainerStarted","Data":"8f8a2ef65ea4414795645e0e3b0b3854ced756756e6983762f601ab6838f86f4"}
Nov 02 02:41:20.650960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:20.650790    4912 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl" event={"ID":"c404169e-f460-414e-8dee-defda48040e1","Type":"ContainerStarted","Data":"a9952003981c3ae11c02e625adce39f1d82a27abf91371b5ddc494c6fa9aa699"}
Nov 02 02:41:20.650960 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:20.650845    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:20.767532 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:20.767484    4912 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl" podStartSLOduration=1.767464003 podStartE2EDuration="1.767464003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:41:20.765240411 +0000 UTC m=+4173.670705771" watchObservedRunningTime="2026-11-02 02:41:20.767464003 +0000 UTC m=+4173.672929396"
Nov 02 02:41:21.176355 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:21.176323    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pvcfb_2a3f2067-4ebd-4f7b-92c6-89715e73ac90/manager/0.log"
Nov 02 02:41:26.633320 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:26.633287    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-gbxrn_ce9b8562-0baf-437e-bf85-5cc8f0e32de1/cluster-monitoring-operator/0.log"
Nov 02 02:41:26.668318 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:26.668291    4912 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-jd2rl"
Nov 02 02:41:31.719273 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:31.719238    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvzz9_b938abbc-9101-4d8e-9f4e-f07e99ddc3a2/node-exporter/0.log"
Nov 02 02:41:32.137357 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:32.137328    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvzz9_b938abbc-9101-4d8e-9f4e-f07e99ddc3a2/kube-rbac-proxy/0.log"
Nov 02 02:41:32.554031 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:32.553962    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvzz9_b938abbc-9101-4d8e-9f4e-f07e99ddc3a2/init-textfile/0.log"
Nov 02 02:41:42.112153 ip-10-0-135-33 kubenswrapper[4912]: I1102 02:41:42.112116    4912 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485vxgl6_4af3acb7-dc87-454d-b1de-70d65476d689/istio-proxy/0.log"
Nov 02 02:41:42.137752 ip-10-0-135-33 kubenswrapper[4912]: E1102 02:41:42.137718    4912 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtdwx" containerName="extractor"